Dialogue system, vehicle having the same and dialogue service processing method

ABSTRACT

A dialogue system is provided to assist a user while minimizing distraction and achieve safe driving by adjusting a level of a dialogue service based on a dialogue with the user in a vehicle driving environment and multiple kinds of information including vehicle state information, driving environment information, and user information, and a vehicle having the dialogue system and a dialogue service processing method is provided.

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims priority to Korean Patent Application No.10-2017-0132913, filed on Oct. 13, 2017 in the Korean IntellectualProperty Office, the entire contents of which is incorporated herein forall purposes by this reference.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a dialogue system for ascertaining anintention of a user through a dialogue with the user and supplyinginformation or a service needed by the user, a vehicle having thedialogue system, and a dialogue service processing method.

Description of Related Art

Since vehicle AVNs and most mobile devices have small screens and smallbuttons, inconvenience can occur when visual information is supplied tousers or inputs of users are received.

When a user looks away from the road or takes his or her hands off ofthe steering wheel to confirm visual information or operate units whiledriving, it may pose a risk to safe driving.

Accordingly, when dialogue systems for ascertaining the intentions ofusers through dialogues with the users and supplying services needed bythe users are applied to vehicles, services are expected to be suppliedmore safely and conveniently.

On the other hand, there is concern of the dialogues between the usersand the dialogue systems distracting the users from driving, and it isnecessary for the dialogue systems to inform the users of the necessityto focus on driving based on input driving environments and adjustservice levels.

The information included in this Background of the Invention section isonly for enhancement of understanding of the general background of theinvention and may not be taken as an acknowledgement or any form ofsuggestion that this information forms the prior art already known to aperson skilled in the art.

BRIEF SUMMARY

Various aspects of the present invention are directed to providing adialogue system that assists a user while minimizing distraction andachieves safe driving by adjusting a level of a dialogue service basedon a dialogue with the user in a vehicle driving environment andmultiple kinds of information including vehicle state information,driving environment information, and user information, and provide avehicle having the dialogue system and a dialogue service processingmethod.

Additional aspects of the disclosure will be set forth in part in thedescription which follows and, in part, will be obvious from thedescription, or may be learned by practice of the disclosure.

In accordance with one aspect of the present invention, a dialoguesystem may include: an input processor configured to extract situationinformation including at least one of vehicle state informationregarding a state of a vehicle and driving environment informationregarding a driving environment of the vehicle or an actioncorresponding to speech of a user; a storage configured to store thecollected situation information; a dialogue manager configured todetermine a service level to be supplied based on the situationinformation and determine an action of adjusting the determined servicelevel; and a result processor configured to generate speech to executethe determined action and generate a control signal of servicelimitation based on the determined service level.

The storage may store a dialogue policy to be executed in accordancewith the determined service level. The result processor may generate aresponse for a result of the service limitation based on the dialoguepolicy.

The storage may store situation information associated with theextracted action. The input processor may extract an action of servicelevel adjustment associated with an action corresponding to the speech.

The input processor may request the vehicle to transmit situationinformation necessary to determine the service level when situationinformation regarding the action for the service level adjustmentassociated with the action corresponding to the speech is not stored inthe storage.

The dialogue manager may acquire situation information necessary todetermine whether an action corresponding to the determined servicelevel is executed from the storage and determine an action of adjustingthe service level based on the acquired situation information.

The storage may store a determination parameter for determining theservice level.

The dialogue manager may determine the service level based on theparameter and the situation information.

The dialogue manager may determine an action of readjusting the adjustedservice level based on new situation information received by the inputprocessor.

The result processor may generate speech to execute an action regardingthe changed service level.

The result processor may generate a control signal of the vehicle basedon the changed service level.

The dialogue system may further include a communication deviceconfigured to receive the situation information from the vehicle andtransmit a response to the vehicle.

The dialogue system may further include a communication deviceconfigured to communicate with an external server. The dialogue managermay request the external server to transmit a factor value when thesituation information is not able to be acquired from the storage.

The dialogue system may further include a communication deviceconfigured to receive the situation information from a mobile deviceconnected to the vehicle and transmit a response and the control signalof the service limitation to the mobile device.

In accordance with another aspect of the present invention, a dialogueservice processing method may include: storing situation informationincluding at least one of vehicle state information regarding a state ofa vehicle and driving environment information regarding a drivingenvironment of the vehicle in a storage; extracting the situationinformation and an action corresponding to speech of a user; acquiring,from the storage, at least one of a factor value of a conditiondetermination factor used to determine at least one action candidateincluding the action corresponding to the speech and a factor value ofan action factor used to execute the at least one action candidate;determining a service level to be supplied based on the situationinformation and an action to be executed from the at least one actioncandidate based on the acquired factor value of the conditiondetermination factor; and generating a control signal of servicelimitation based on the determined service level and a response toexecute the determined action using the acquired factor value of theaction factor.

The storage may store a dialogue policy to be executed in accordancewith the service level. The generating of the control signal may includegenerating a response for a result of the service limitation based onthe dialogue policy.

The dialogue service processing method may further include: storingsituation information regarding an action in the storage by action; andacquiring situation information regarding the action corresponding tothe speech from the storage and transmitting the situation informationto a dialogue manager.

The dialogue service processing method may further include requestingthe vehicle to transmit situation information necessary to determine theservice level when the situation information regarding the actioncorresponding to the speech is not stored in the storage.

The determining of the service level may include determining the servicelevel using the requested situation information as the factor value ofthe condition determination factor or the factor value of the actionfactor.

The service level may be changed based on the situation informationdelivered by the vehicle.

The situation information may include at least one of navigationinformation, accident information, and weather information.

In accordance with yet another aspect of the present invention, avehicle may include: a dialogue system including an input processorconfigured to extract situation information including at least one ofvehicle state information regarding a state of a vehicle and drivingenvironment information regarding a driving environment of the vehicleor an action corresponding to speech of a user, a storage configured tostore the collected situation information, a dialogue manager configuredto determine a service level to be supplied based on the situationinformation and determine an action of adjusting the determined servicelevel, and a result processor configured to generate speech to executethe determined action and generate a control signal of servicelimitation based on the determined service level.

The vehicle may further include a controller configured to execute anoperation related to the determined service based on the control signal.

According to the dialogue system, a vehicle having the dialogue system,and the dialogue service processing method according to various aspectsof the present invention, it is possible to minimize distraction to auser and achieve safe driving by adjusting a level of a dialogue servicebased on a dialogue with the user in a vehicle driving environment andmultiple kinds of information including vehicle state information,driving environment information, and user information.

The methods and apparatuses of the present invention have other featuresand advantages which will be apparent from or are set forth in moredetail in the accompanying drawings, which are incorporated herein, andthe following Detailed Description, which together serve to explaincertain principles of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a control block diagram of a dialogue system according to anexemplary embodiment;

FIG. 2 is a view illustrating an internal configuration of a vehicle;

FIG. 3, FIG. 4, and FIG. 5 are views illustrating an exemplary dialogueexchanged between the dialogue system and a driver;

FIG. 6 and FIG. 7 are control block diagrams illustrating a connectionrelation among constituent elements of the dialogue system and thevehicle in brief;

FIG. 8 and FIG. 9 are control block diagrams illustrating a connectionrelation among constituent elements of the dialogue system and thevehicle in brief;

FIG. 10 is a control block diagram illustrating a vehicle-independentscheme in which the dialogue system is provided in the vehicle;

FIG. 11 and FIG. 12 are control block diagrams illustrating a vehiclegateway scheme in which the dialogue system is provided in a remoteserver and the vehicle is configured as only a gateway connecting a userto the dialogue system;

FIG. 13 is a control block diagram for a case in which the vehicle mayexecute parts of an input process and an output process in the vehiclegateway scheme;

FIG. 14 is a control block diagram illustrating a hybrid scheme in whichboth a remote dialogue system server and the vehicle may executedialogue processing;

FIG. 15 and FIG. 16 are control block diagrams illustrating a mobilegateway scheme in which a mobile device connected to the vehicleconnects a user to the remote dialogue system server;

FIG. 17 is a control block diagram illustrating a mobile-independentscheme in which the dialogue system is mounted on a mobile device;

FIG. 18, FIG. 19A, and FIG. 19B are control block diagrams illustratinga subdivided configuration of an input processor in the configuration ofthe dialogue system;

FIG. 20A and FIG. 20B are views illustrating exemplary informationstored in a situation understanding table;

FIG. 21 is a control block diagram illustrating a subdividedconfiguration of a dialogue manager;

FIG. 22 is a view illustrating exemplary information stored in anassociated action DB;

FIG. 23 is a view illustrating exemplary information stored in an actionexecution condition DB;

FIG. 24 is a view illustrating exemplary information stored in an actionfactor DB;

FIG. 25 is a view illustrating exemplary information stored in anambiguity resolution information DB;

FIG. 26A and FIG. 26B are tables summarizing various examples in whichan ambiguity resolver resolves ambiguity, extracts an action withreference to the ambiguity resolution information DB, and controls avehicle;

FIG. 27 is a control block diagram illustrating a subdividedconfiguration of a result processor;

FIG. 28 is a flowchart illustrating adjustment of a service level by thedialogue system according to an exemplary embodiment;

FIG. 29 is a view illustrating a determination reference by which aservice level is determined according to an exemplary embodiment;

FIG. 30 is a view illustrating an example of limitation of a serviceadjusted in accordance with a service level; and

FIG. 31 is a flowchart illustrating adjustment of a service levelaccording to another exemplary embodiment of the present invention.

It may be understood that the appended drawings are not necessarily toscale, presenting a somewhat simplified representation of variousfeatures illustrative of the basic principles of the invention. Thespecific design features of the present invention as included herein,including, for example, specific dimensions, orientations, locations,and shapes will be determined in part by the particular intendedapplication and use environment.

In the figures, reference numbers refer to the same or equivalent partsof the present invention throughout the several figures of the drawing.

DETAILED DESCRIPTION

Reference will now be made in detail to various embodiments of thepresent invention(s), examples of which are illustrated in theaccompanying drawings and described below. While the invention(s) willbe described in conjunction with exemplary embodiments, it will beunderstood that the present description is not intended to limit theinvention(s) to those exemplary embodiments. On the contrary, theinvention(s) is/are intended to cover not only the exemplaryembodiments, but also various alternatives, modifications, equivalentsand other embodiments, which may be included within the spirit and scopeof the invention as defined by the appended claims.

Like reference numerals refer to like elements throughout thespecification. Not all details of embodiments of the present inventionare described herein, and description of general art to which thepresent invention pertains and overlapping descriptions betweenembodiments are omitted. Components indicated by terms including “unit,”“module,” “member,” and “block” herein may be implemented by software orhardware. According to different embodiments, a plurality of units,modules, members, and blocks may be implemented by a single element, oreach of a single unit, a single module, a single member, and a singleblock may include a plurality of elements.

Throughout the specification, a certain part being “connected” toanother part includes the certain part being directly connected to theother part or being indirectly connected to the other part. Indirectconnection includes being connected through a wireless communicationnetwork.

Also, a certain part “including” a certain element signifies that thecertain part may further include another element instead of excludingother elements unless particularly indicated otherwise.

Singular expressions include plural referents unless clearly indicatedotherwise in context.

A reference numeral provided to each step for convenience of descriptionis used to identify each stage. The reference numerals are not fordescribing an order of the steps, and the steps may be performed in anorder different from that shown in the drawings unless a specific orderis clearly described in the context.

Hereinafter, embodiments of a dialogue system, a vehicle having thesame, and a dialogue service processing method will be described morespecifically with reference to the accompanying drawings.

The dialogue system according to one exemplary embodiment is anapparatus that ascertains an intention of a user using a voice of theuser and an input other than the voice and supplies a serviceappropriate for the intention of the user or a service needed by theuser. The dialogue system can execute a dialogue with the user byoutputting system speech as means for supplying a service or means forclearly ascertaining the intention of the user.

According to various aspects of the present invention, services suppliedto a user may include all the operations executed to meet a necessity ofthe user or an intention of the user including supplying information,controlling a vehicle, executing audio/video/navigation functions, andsupplying content delivered from an external server.

The dialogue system according to one exemplary embodiment can accuratelyascertain an intention of a user in a special environment including avehicle by providing a dialogue processing technology specified for avehicle environment.

A gateway connecting the dialogue system to a user may be a vehicle or amobile device connected to the vehicle. As will be described below, thedialogue system may be provided in a vehicle or may be provided in anoutside remote server of a vehicle to exchange data throughcommunication with the vehicle or a mobile device connected to thevehicle.

Some of the constituent elements of the dialogue system may be providedin a vehicle and some of the constituent elements may be provided in aremote server so that an operation of the dialogue system can also beexecuted partially in the vehicle and the remote server.

FIG. 1 is a control block diagram of a dialogue system according to anexemplary embodiment.

Referring to FIG. 1, a dialogue system 100 according to an exemplaryembodiment of the present invention includes an input processor 110configured to process a user input including a voice of the user and aninput other than the voice or an input including information regarding avehicle or information regarding the user, a dialogue manager 120configured to ascertain an intention of the user using a processingresult of the input processor 110 and determine an action correspondingto the intention of the user or a state of the vehicle, a resultprocessor 130 configured to supply a specific service in accordance withan output result of the dialogue manager 120 or output system speech tocontinue a dialogue, and a storage 140 configured to store variousinformation necessary for the dialogue system 100 to execute anoperation to be described.

The input processor 110 may receive two types of inputs, a voice of theuser and an input other than a voice. The input other than a voice mayinclude not only recognition of a user's gesture and non-voice of theuser input through an operation on an input device but also vehiclestate information indicating a state of a vehicle, driving environmentinformation regarding a vehicle driving environment, and userinformation indicating a state of the user. In addition to theinformation, all the information regarding the vehicle and the user maybe input to the input processor 110 as long as the information isinformation that can be used to ascertain an intention of the user orsupply a service to the user. The user may include both a driver and apassenger.

The input processor 110 recognizes an input user's voice, converts thevoice into a spoken sentence in a text format, and ascertains anintention of the user by applying a natural language understandingtechnology to the spoken sentence of the user.

The input processor 110 collects information regarding a state of thevehicle or a driving environment other than the user's voice andunderstands a situation using the collected information.

The input processor 110 delivers information regarding an intention anda situation of the user ascertained through the natural languageunderstanding to the dialogue manager 120.

The dialogue manager 120 determines an action corresponding to theintention of the user and a current situation based on the informationregarding the intention of the user and the situation delivered from theinput processor 110 and manages factors necessary to execute the action.

The action according to the embodiment may be any of the operationsexecuted to supply a specific service and the kinds of actions may bedefined in advance. In some cases, supplying a service and executing anaction may have the same meaning.

For example, actions including service level adjustment, road guidance,vehicle state inspection, and gas station recommendation may be definedin advance in a domain/action inference rule DB 141 (see FIG. 19A), andan action corresponding to speech of the user, that is, an actionintended by the user, may be extracted from actions defined in advancein accordance with stored inference rules. Actions associated withevents occurring in vehicles may be defined in advance and stored in anassociated action DB 146 b (see FIG. 21).

The kinds of actions are not limited. The actions can be anything thatcan be executed through a vehicle 200 or a mobile device 400 by thedialogue system 100 as long as the actions are defined in advance andinference rules or relations with other actions/events are stored.

The dialogue manager 120 delivers information regarding a determinedaction to the result processor 130.

The result processor 130 generates and outputs a dialogue response and acommand necessary to execute the delivered action. The dialogue responsecan be output as text, an image, or audio. When the command is output, aservice including vehicle control and external content supplycorresponding to the output command can be executed.

The storage 140 stores various information necessary for dialogueprocessing and service supply. For example, information regarding adomain, an action, a speech act, and an entity name used for naturallanguage understanding may be stored in advance, a situationunderstanding table used to understand a situation from inputinformation may be stored, or data detected by a detector provided inthe vehicle, information regarding the user, and information necessaryto execute an action may be stored in advance. The information stored inthe storage 140 will be described in more detail later.

As described above, the dialogue system 100 provides a dialogueprocessing technology specialized in a vehicle environment. All or someof the constituent elements of the dialogue system 100 may be includedin the vehicle. The dialogue system 100 may be provided in a remoteserver and the vehicle is configured as only a gateway between thedialogue system 100 and the user. In either case, the dialogue system100 may be connected to the user via the vehicle or a mobile deviceconnected to the vehicle.

FIG. 2 is a view illustrating an internal configuration of a vehicle.

Referring to FIG. 2, a display 231 displaying screens necessary tocontrol the vehicle including an audio function, a video function, anavigation function, or a telephoning function and input buttons 221 toreceive inputs of control commands of the user may be provided in acenter fascia 203 which is a central region of a dashboard 201 withinthe vehicle 200.

To facilitate an operation of a driver, input buttons 223 may beprovided on a steering wheel 207 and a jog shuttle 225 executing a roleof an input button may be provided in a center console region 202between a driver seat 254 a and a passenger seat 254 b.

A module including a processor configured to control a display 231, aninput button 221, and various functions is configured as an audio videonavigation (AVN) terminal or is configured as a head unit.

The display 231 may be realized as one of diverse display unitsincluding a liquid crystal display (LCD), a light emitting diode (LED),a plasma display panel (PDP), an organic light emitting diode (OLED),and a cathode ray tube (CRT).

The input button 221 may be provided as a hard key type in a regionadjacent to the display 231, as illustrated in FIG. 2, and the display231 may also execute a function of the input button 221 when the display231 is realized as a touch screen type.

The vehicle 200 may receive a command of the user as a voice input via avoice input device 210. The voice input device 210 may include amicrophone configured to receive a sound input, convert the sound inputinto an electric signal, and output the electric signal.

To input a voice effectively, the voice input device 210 may be providedin a head lining 205, as illustrated in FIG. 2, but embodiments of thevehicle 200 are not limited thereto. The voice input device 201 may beprovided on the dashboard 201 or may be provided on the steering wheel207. Additionally, there is no limitation as long as the voice inputdevice 201 is located at a position at which a voice of the user isappropriately received.

Within the vehicle 200, a speaker 232 configured to execute a dialoguewith the user or output a sound necessary to provide a service desiredby the user may be provided. For example, the speaker 232 may beprovided inside a driver seat door 253a and a passenger seat door 253b.

The speaker 232 may output a voice for navigation route guidance, asound or a voice included in audio/video content, a voice for providinginformation or a service desired by the user, and system speechgenerated as a response to speech of the user.

The dialogue system 100 according to an exemplary embodiment of thepresent invention may provide a service optimized for a lifestyle of theuser using a dialogue processing technology specialized in a vehicleenvironment and may configure a new service using a technology includinga connected car, an Internet of things (IoT), artificial intelligence(AI), or the like.

When the dialogue processing technology specialized in a vehicleenvironment as in the dialogue system 100 according to an exemplaryembodiment is applied, it is easy to recognize and handle mainsituations (contexts) in a direct driving situation of the driver. Aservice may be provided weighting factors having an influence on drivingincluding flow shortage and drowsiness, and it is easy to acquireinformation necessary to provide a service including a driving time anddestination information, based on a condition during movement to adestination in most cases.

It is easy to realize an intelligent service for ascertaining anintention of the driver and proposing a function. This is becausereal-time information and an action are preferentially considered in adirect driving situation of the driver. For example, when the driversearches for a gas station while driving, the searching may beinterpreted as a driver's intention to approach a gas station now.However, when a gas station is searched for in a non-vehicleenvironment, the searching may be interpreted as multiple intentionsincluding a position information inquiry, a telephone number inquiry,and a price inquiry in addition to an intention to approach a gasstation now.

A vehicle has limited space, but there are various situations inside thevehicle. For example, the dialogue system 100 may be utilized in adriving situation of an unfamiliar interface of a rent vehicle or thelike, a situation of a chauffeur service, a management situationincluding vehicle washing, a situation in which an infant is beingaccommodated, a situation in which a driver visits a specificdestination, and the like.

There are various service opportunities and dialogue situations invehicle driving including a vehicle inspection stage, a departurepreparation stage, a driving stage, and a parking stage, and stagesbefore and after the driving of the vehicle. The dialogue system 100 maybe utilized in situations including a situation in which a driver doesnot know countermeasures of vehicle problems, a linkage situation of avehicle with various external units, a driving habit confirmationsituation of fuel efficiency, a safety support function utilizationsituation including smart cruise control, a navigation operationsituation, a drowsiness situation, a situation in which a driverrepeatedly drives along the same route every day, and a situation inwhich a driver confirms a place at which the driver can park or stop avehicle.

FIG. 3, FIG. 4, and FIG. 5 are views illustrating an exemplary dialogueexchanged between the dialogue system and a driver.

Referring to FIG. 3, when a driver inputs speech (U1: Show me the way toIncheon International Airport) for inquiring about guidance, thedialogue system 100 may recognize a destination and output speech (S1:Starting guide to Incheon International Airport) for informing thedriver of a response to the request to indicate that vehicle controlstarts.

The dialogue system 100 according to an exemplary embodiment includedtogether may also output speech for supplying information to indicatewhether the user has traveled a route to the requested destinationbefore, speech (S2: This is your first time on this route. You shouldlimit use of the dialogue service) for cautioning about dialogue serviceuse, and speech (S3: Would you still like to use it?) for inquiringabout continuing to use the dialogue system.

The driver may input speech (U2: Yes, I'll use it) for responding to theinquiry of the dialogue system 100 with agreement. The dialogue system100 may output speech (S4: I will adjust the service level to help youfocus on driving. Do you agree?) for inquiring of the user aboutagreement with the adjustment of the service level in a response to theuser.

In accordance with speech (U3: Yes) of the agreement of the driver, thedialogue system 100 determines a service level to which adjustment ispossible during the driving on a new route based on data set in advanceor received from the outside.

The dialogue system 100 may output speech (S5: The service level ischanged to attention stage 1) for notifying the driver that a serviceadjusted in accordance with the finally determined service level isoutput.

Referring to FIG. 4, the dialogue system 100 may receive vehicle drivingenvironment information indicating that the vehicle 200 enters a safedriving caution section from a controller 240 during the driving. Basedon the vehicle driving environment information, the dialogue system 100determines that it is necessary to adjust the service level through thefollowing speeches.

The dialogue system 100 may output speech (S1: You are now in a safedriving caution section) for supplying information regarding safedriving and may output speech (S2: You need to drive cautiously, andthus I recommend that you not use this until later. Would you still liketo use it?) for inquiring about continuing to use the dialogue system.

The driver may input speech (U1: Yes, I'll use it) for responding to theinquiry of the dialogue system 100 with agreement. The dialogue system100 may output speech (S3: The service level is adjusted to focus ondriving. Do you agree?) for inquiring of the user about agreement of theadjustment of the service level in a response to the user.

In accordance with a speech (U2: Yes) of the agreement of the driver,the dialogue system 100 determines a service level adjustable during thedriving of the safe driving caution section based on data set in advanceor received from the outside.

The dialogue system 100 may output a speech (S4: I will adjust theservice level is changed to attention stage 2) for notifying the driverthat a service adjusted in accordance with the finally determinedservice level is output.

Referring to FIG. 5, the dialogue system 100 may receive vehicle drivingenvironment information indicating that a section in which fogfrequently occurs and accident information on a driving route have beencollected during the driving of the vehicle 200 from the outside thereofvia the controller 240 or a communication device 280. Based on thevehicle driving environment information, the dialogue system 100determines that it is necessary to adjust the service level through thefollowing speech.

The dialogue system 100 may output speech (S1: A section in which fogfrequently occurs and accident information on a driving route have justbeen collected) for supplying information regarding safe driving and mayoutput speech (S2: You need to drive cautiously, and thus I recommendthat you not use this until later. Would you still like to use it) forinquiring about continuing to use the dialogue system.

Unlike FIG. 4, the driver may omit speech for responding to the inquirywith agreement and immediately input speech (U1: Adjust the servicelevel) for requesting the dialogue system to adjust the service level.

The dialogue system 100 may analyze the dialogue requested by the userand output speech (S3: Okay, the service level is changed to attentionstage 3) for informing the driver of a response to the adjustment of theservice level and an execution result of the adjustment result.

In the present way, the dialogue system 100 may adjust the level of thedialogue service by also considering content not spoken by the userbased on vehicle driving information. Thus, the dialogue system 100 mayimprove dialogue efficiency and a safe driving possibility by adjustinga service desired by the user depending on a situation.

FIG. 6 and FIG. 7 are control block diagrams illustrating a connectionrelation among constituent elements of the dialogue system and thevehicle in brief

Referring to FIG. 6, a user's voice input to the dialogue system 100 maybe input via the voice input device 210 provided in the vehicle 200. Asdescribed above in FIG. 2, the voice input device 210 may include amicrophone provided within the vehicle 200.

During an input by the user, an input other than a voice may be inputvia a non-voice input device 220. The non-voice input device 220 mayinclude the input buttons 221 and 223 receiving commands through a useroperation and the jog shuttle 225.

The non-voice input device 220 may also include a camera configured toimage the user. A gesture, an expression, or a visual line direction ofthe user used as means for a command input may be recognized through avideo captured by the camera. Alternatively, a state (drowsy state orthe like) of the user may also be ascertained through a video capturedby the camera.

Information regarding a vehicle may be input to the dialogue system 100via the controller 240. The information regarding a vehicle may includevehicle state information or surrounding situation information acquiredthrough various detectors provided in the vehicle 200 and may alsoinclude information basically stored in the vehicle 200 including a typeof fuel of the vehicle.

The dialogue system 100 ascertains an intention and a situation of theuser using a user's voice input via the voice input device 210, anon-voice input of the user input via the non-voice input device 220,and various information input via the controller 240 and outputs aresponse to execute an action corresponding to the intention of theuser.

A dialogue partner output device 230 is a device configured to supply avisual, auditory, or tactile output to a dialogue partner and mayinclude the display 231 and the speaker 232 provided in the vehicle 200.The display 231 and the speaker 232 may visually or auditorily output aresponse to speech of the user, an inquiry about the user, orinformation requested by the user. Alternatively, a vibrator may bemounted on the steering wheel 207 to output vibration.

The controller 240 may control the vehicle 200 to execute an actioncorresponding to an intention or a current situation of the user inaccordance with a response output from the dialogue system 100.

On the other hand, the vehicle 200 may collect not only data acquiredthrough detectors provided in the vehicle 200 but also informationacquired from an external content server 300 or an external device viathe communication device 280, for example, driving environmentinformation and user information including a traffic situation, weather,a temperature, passenger information, and driver personal information,and may deliver the information to the dialogue system 100.

As illustrated in FIG. 7, information including a remaining fuel amount,a rainfall amount, a rainfall speed, nearby obstacle information, aspeed, engine temperature, a tire air pressure, and a current position,acquired from detectors provided in the vehicle may be input to thedialogue system 100 via an internal signal controller 241.

Driving environment information acquired from the outside thereofthrough vehicle-to-everything (V2X) may be input to the dialogue system100 via an external signal controller 242. V2X means that various usefulinformation including a traffic situation is exchanged or shared while avehicle is travelling and mutual communication with other vehicles androad infrastructures is performed.

V2X communication may include vehicle-to-infrastructure (V2I)communication, vehicle-to-vehicle (V2V) communication, andvehicle-to-nomadic-units (V2N) communication. Accordingly, when the V2Xcommunication is used, direct communication between vehicles orcommunication with infrastructures disposed in streets may be performedand information regarding front traffic information, approach of othervehicles, or a collision possibility may be exchanged to be reported toa driver.

Accordingly, the driving environment information input to the dialoguesystem 100 via the external signal controller 242 may include fronttraffic information, approach information related to surroundingvehicles, a collision warning with other vehicles, a real-time trafficsituation, an unexpected incident, and a traffic flow control state.

Although not illustrated, a signal acquired through V2X may also beinput to the vehicle 200 via the communication device 280.

The controller 240 may include a memory configured to store programsused to execute the above-described operations or operations to bedescribed below and a processor configured to execute the storedprograms. A single memory and a single processor may be provided. Whenpluralities of memories and processors are provided, the memories andthe processors may be integrated on one chip or may be physicallydivided.

The internal signal controller 241 and the external signal controller242 may be realized by the same processor or may be realized by separateprocessors.

FIG. 8 and FIG. 9 are control block diagrams illustrating a connectionrelation among constituent elements of the dialogue system and thevehicle in brief

Referring to FIG. 8, a voice of a user transmitted from the voice inputdevice 210 is input to a voice input processor 111 in the inputprocessor 110, and a non-voice input of the user transmitted from thenon-voice input device 220 is input to a situation information processor112 in the input processor 110.

The situation information processor 112 understands a situation based onthe vehicle state information, the driving environment information, andthe user information. The dialogue system 100 may ascertain an intentionof the user more accurately through such situation understanding or mayeffectively search for a service currently needed by the user.

A response output from the result processor 130 may be input to thedialogue partner output device 230 or the controller 240 so that thevehicle 200 can supply a service needed by the user. Alternatively, theresponse may be transmitted to the external content server 300 torequest a service needed by the user.

The vehicle state information, the driving environment information, andthe user information transmitted from the controller 240 are stored inthe storage 140.

Referring to FIG. 9, the storage 140 may include a long-term memory 143and a short-term memory 144. The data stored in the storage 140 may beseparately stored in the short-term and long-term memories depending onimportance of the data, permanence of the data, and an intention of adesigner.

The short-term memory 144 may store previously executed dialogues. Theprevious executed dialogues may be dialogues executed within a referencetime from a current time. Alternatively, dialogues may be storedcontinuously until capacity of speech content between the user and thedialogue system 100 reaches a reference value.

For example, when a user says “Tell me restaurants near GangnamStation,” the dialogue system 100 searches for restaurants near GangnamStation through the external content server 300 and supplies informationregarding the located restaurants near Gangnam Station to the user. Asan example of supply of information, a restaurant list may be displayedon the display 231. When a user says “The first one,” dialogue contentfrom an inquiry to selection of a restaurant may be stored in theshort-term memory 144.

Alternatively, the whole dialogue content may be stored and specificinformation included in the dialogue content may also be stored. Forexample, the first restaurant in the restaurant list may also be storedas the restaurant selected by the user in the short-term memory 144 orthe long-term memory 143.

When the user asks the dialogue system 100 “What about the weather?”after the dialogue about the restaurants near Gangnam Station, thedialogue system 100 may infer that a position of interest of the user isGangnam Station from the dialogue stored in the short-term memory 144and may output a response “It is raining near Gangnam Station,” to theuser.

Thereafter, when the user says “What do you recommend at therestaurant,” the dialogue system 100 may infer that the “restaurant” isthe restaurant near Gangnam Station from the dialogue stored in theshort-term memory, may acquire information regarding a recommended dishat the restaurant through a service supplied from the external contentserver 300, and may output a response including “The noodle soup is goodat that restaurant.”

The long-term memory 143 may store data depending on whether data ispermanent. For example, it may be determined that permanence of data inregard to telephone numbers of family or friends, POI information suchas home or a company, and preference of the user for specific factors isguaranteed, and thus the data may be stored in the long-term memory 143.Conversely, data for which it is determined that the permanence of thedata is not guaranteed may be stored in the short-term memory 144.

For example, since a current position of the user is temporary data, thecurrent position of the user may be stored in the short-term memory 144.Since a destination included in a road guide request can be viewed aspermanent data serving as a reference of determination regarding whetherthe user is visiting the destination for the first time, the destinationmay be stored in the long-term memory 144.

When a user says “Show me the way to Incheon International Airport,” thedialogue system 100 may ascertain a current position of the user fromthe short-term memory 144 and may determine whether there is dataregarding Incheon International Airport or a destination related toIncheon International Airport from the long-term memory 143.

The dialogue system 100 may also supply a service or informationpreemptively to the user using data stored in the long-term memory 143and the short-term memory 144.

For example, the long-term memory 143 may store service levels suppliedby the dialogue system 100 and kinds of functions limited in thedialogue system in accordance with the service levels. When the dialoguesystem 100 receives weather information indicating that driving isdifficult from the vehicle controller 240, the dialogue system 100 maysupply agreement or disagreement of service level adjustment dependingon a service level adjustment reference stored in the long-term memory143.

As another example, information regarding weather may be stored in theshort-term memory 144. When the driving of the vehicle ends, informationregarding weather is received from the outside of the vehicle andbecomes meaningless in many cases. The dialogue system 100 may analyzethe information regarding weather stored in the short-term memory 144and may output “I have information that there will be fog on your routeshortly. I recommend that you not use the dialogue system is until laterso that you can drive safely.”

FIG. 10 is a control block diagram illustrating a vehicle-independentscheme in which the dialogue system is provided in the vehicle.

In the vehicle-independent scheme, as illustrated in FIG. 10, thedialogue system 100 including the input processor 110, the dialoguemanager 120, the result processor 130, and the storage 140 may beincluded in the vehicle 200.

When the dialogue system 100 is disposed in the vehicle 200, the vehicle200 may autonomously process a dialogue with a user and supply a neededservice to the user. Here, information necessary for dialogue processingand service supply may be taken from the external content server 300.

The vehicle state information or the driving environment informationincluding a remaining fuel amount, a rainfall amount, a rainfall speed,nearby obstacle information, a tire air pressure, a current position, anengine temperature, and a vehicle speed, detected by a vehicle detector260 is input to the dialogue system 100 via the controller 240.

The controller 240 may control an air conditioning device 251, a window252, a door 253, a seat 254, or an AVN 255 provided in the vehicle 200in accordance with a response output by the dialogue system 100.

For example, the dialogue system 100 may determine that an intention ofa user or a service needed by the user is lowering of an internaltemperature of the vehicle 200. When a command corresponding to thedetermination is generated and output, the controller 240 may controlthe air conditioning device 251 to lower an internal temperature of thevehicle 200.

As another example, the dialogue system 100 may determine that anintention of the user or a service needed by the user is raising adriver seat window 252 a . When a command corresponding to thedetermination is generated and output, the controller 240 may controlthe window 252 to raise the driver seat window 252 a.

As yet another example, the dialogue system 100 may determine that aservice level corresponding to an intention of a user is adjusted. Thedialogue system 100 may give a command for a control operation of thecontroller 240 supplied in cooperation with the dialogue system 100 inaccordance with the service level and the controller 240 may interrupt arelated control operation.

FIG. 11 and FIG. 12 are control block diagrams illustrating a vehiclegateway scheme in which the dialogue system is provided in a remoteserver and the vehicle is configured as only a gateway connecting a userto the dialogue system.

In the vehicle gateway scheme, as illustrated in FIG. 11, a remotedialogue system server 1 is provided outside of the vehicle 200 and adialogue system client 270 connected to the remote dialogue systemserver 1 via the communication device 280 is provided in the vehicle200. The communication device 280 is configured as a gateway connectingthe vehicle 200 to the remote dialogue system server 1.

The dialogue system client 270 may function as an interface connected toan input and output device and execute collection and transmission orreception of data.

When the voice input device 210 and the non-voice input device 220provided in the vehicle 200 receive user inputs and deliver the userinputs to the dialogue system client 270, the dialogue system client 270may transmit input data to the remote dialogue system server 1 via thecommunication device 280.

The controller 240 may also deliver data detected by the vehicledetector 260 to the dialogue system client 270, and the dialogue systemclient 270 can deliver data detected by the vehicle detector 260 to theremote dialogue system server 1 via the communication device 280.

The above-described dialogue system 100 may be provided in the remotedialogue system server 1, so that a process for input data, dialogueprocessing based on a processing result of input data, and resultprocessing based on a dialogue processing result can all be executed.

The remote dialogue system server 1 may take information or contentnecessary for the process for the input data, dialogue management, orresult processing from the external content server 300.

The vehicle 200 may also take content necessary to supply a serviceneeded by the user in accordance with a response transmitted from theremote dialogue system server 1 from the external content server 300.

Referring to FIG. 12, the communication device 280 may include one ormore communication modules configured for communicating with an externalunit and may include, for example, a short-range communication module281, a wired communication module 282, and a wireless communicationmodule 283.

The short-range communication module 281 may include at least one ofvarious short-range communication modules configured to transmit andreceive signals at a short distance using a wireless communicationnetwork including a Bluetooth module, an infrared communication module,a radio frequency identification (RFID) communication module, a wirelesslocal access network (WLAN) communication module, an NFC communicationmodule, a Zigbee communication module.

The wired communication module 282 may include at least one of not onlyvarious wired communication modules including a local area network (LAN)module, a wide area network (WAN) module, and a value added network(VAN) module but also various cable communication modules includingUniversal Serial Bus (USB), High Definition Multimedia Interface (HDMI),Digital Visual Interface (DVI), Recommended Standard (RS)-232, powerline communication, and Plain Old Telephone Service (POTS).

The wireless communication module 283 may include at least one of notonly a WiFi module and a wireless broadband (WiBro) module but alsovarious wireless communication modules connectable to an Internetnetwork in a wireless communication scheme including Global System forMobile Communication (GSM), Code Division Multiple Access (CDMA),Wideband Code Division Multiple Access (WCDMA), Universal MobileTelecommunications System (UMTS), Time Division Multiple Access (TDMA),and Long Term Evolution (LTE), 4G, and 5G.

On the other hand, the communication device 280 may further include aninternal communication module for communication between internalelectronic devices of the vehicle 200. As an internal communicationprotocol of the vehicle 200, controller area network (CAN), localinterconnection network (LIN), FlexRay, or Ethernet may be used.

The dialogue system 100 may exchange data with the external contentserver 300 or the remote dialogue system server 1 using the wirelesscommunication module 283. V2X communication may be executed using thewireless communication module 283. Data may be exchanged with a mobiledevice connected to the vehicle 200 using the short-range communicationmodule 281 or the wired communication module 282.

FIG. 13 is a control block diagram for a case in which the vehicle mayexecute parts of an input process and an output process in the vehiclegateway scheme.

As described above, the dialogue system client 270 of the vehicle 200may execute only functions of collecting and transmitting or receivingdata. As illustrated in FIG. 13, the input processor 271, the resultprocessor 273, and the storage 274 may also be included in the dialoguesystem client 270 and a process for data input from the user or thevehicle or a process related to supply of a service determined to beneeded by the user may also be executed in the vehicle 200. That is,operations of the input processor 110 and the result processor 130 maybe executed not only in the remote dialogue system server 1 but also inthe vehicle 200.

In the instant case, the operation of the above-described inputprocessor 110 may be executed partially and may also be executed whollyby the dialogue system client 270. An operation of the above-describedresult processor 130 may also be executed partially or may also beexecuted wholly by the dialogue system client 270.

A task burden between the remote dialogue system server 1 and thedialogue system client 270 may be determined in consideration of thecapacity of data to be processed, a data processing speed, or the like.

On the other hand, the dialogue system client 270 according to anexemplary embodiment of the present invention may also determine aservice limitation level based on the driving environment informationdelivered by the controller 240. Here, in the instant case, the dialoguesystem client 270 may deliver the driving environment information to theremote dialogue system server 1 via the communication device 280, andmay limit an output in accordance with a service level determined by theremote dialogue system server 1 or may control an operation of thecontroller 240.

FIG. 14 is a control block diagram illustrating a hybrid scheme in whichboth a remote dialogue system server and the vehicle may execute adialogue process.

In the hybrid scheme, as illustrated in FIG. 14, the remote dialoguesystem server 1 may also include the input processor 110, the dialoguemanager 120, the result processor 130, and the storage 140 and mayexecute dialogue processing. The vehicle 200 may also include a terminaldialogue system 290 configured to include an input processor 291, adialogue manager 292, a result processor 293, and a storage 294 and mayexecute dialogue processing.

However, the processor or the memory provided in the vehicle 200 may bedifferent from the processor or the memory provided in the remotedialogue system server 1 in terms of capacity or performance.Accordingly, when the terminal dialogue system 290 may process all inputdata, manage a dialogue, and output a result, the terminal dialoguesystem 290 may execute the whole process. Otherwise, the remote dialoguesystem server 1 may be requested to execute the process.

Before executing dialogue processing, the terminal dialogue system 290may determine whether to execute a process of the terminal dialoguesystem 290 based on a kind of input data and/or may directly execute theprocess in accordance with a determined result or request the remotedialogue system server 1 to execute the process.

Alternatively, when a situation in which the dialogue processing may notbe executed occurs while the terminal dialogue system 290 executes thedialogue processing, the terminal dialogue system 290 may transmit aprocessed result while requesting the remote dialogue system server 1 toexecute the process.

For example, when high-performance computing power and long-term dataprocessing are necessary, the remote dialogue system server 1 mayexecute a process and the terminal dialogue system 290 may execute aprocess which has to be executed in real time. For example, when aninstance in which it is necessary to immediately execute a processoccurs and data has to be processed before synchronization, the terminaldialogue system 290 may be set to preferentially process the data.

When an unregistered speaker is in the vehicle and it is necessary toconfirm a user, the remote dialogue system server 1 may be configured toprocess a dialogue.

When the terminal dialogue system 290 may not autonomously completedialogue processing when connection to the remote dialogue system 1 viathe communication device 280 may not be made, the user may be informedof the fact that the dialogue processing may not be executed via thedialogue partner output device 230.

Data to be stored in the terminal dialogue system 290 or data to bestored in the remote dialogue system server 1 may be determined inaccordance with a reference including capacitance of data or a kind ofdata. For example, data for which there is a concern of privacy invasionbecause of personal identification may be stored in a storage 294 of theterminal dialogue system 290. Large-sized data may be stored in thestorage 140 of the remote dialogue system server 1 and small-sized datamay be stored in the storage 294 of the terminal dialogue system 290.Alternatively, small-sized data may also be stored in both the storage140 of the remote dialogue system server 1 and the storage 294 of theterminal dialogue system 290.

On the other hand, the terminal dialogue system 290 according to anexemplary embodiment of the present invention may also determine aservice limitation level based on the driving environment informationdelivered by the controller 240. Here, in the instant case, the terminaldialogue system 290 may deliver the driving environment information tothe remote dialogue system server 1 via the communication device 280 andlimit an output in accordance with a service level determined by theremote dialogue system server 1, or may control an operation of thecontroller 240.

FIG. 15 and FIG. 16 are control block diagrams illustrating a mobilegateway scheme in which a mobile device connected to the vehicleconnects a user to the remote dialogue system server.

In the mobile gateway scheme, as illustrated in FIG. 15, the mobiledevice 400 receives the vehicle state information and the drivingenvironment information from the vehicle 200 and transmits a user inputand vehicle state information to the remote dialogue system server 1.That is, the mobile device 400 is configured as a gateway connecting theuser to the remote dialogue system server 1 or connecting the vehicle200 to the remote dialogue system server 1.

The mobile device 400 may be an electronic device including asmartphone, a smartwatch, smart glasses, a personal digital assistant(PDA), or a tablet PC configured to be portable and configured tocommunicate with an external server and a vehicle to exchange data.

The mobile device 400 includes a voice input device 410 configured toreceive an input of a user's voice, a non-voice input device 420configured to receive a user's non-voice input, an output device 430configured to output a response in a visual, auditory, or tactilemanner, a communication device 480 configured to communicate with theremote dialogue system server 1 and the vehicle 200 to transmit andreceive data, and a dialogue system client 470 configured to collectinput data from the vehicle 200 and a user and transmit the input datato the remote dialogue system server 1 via the communication device 480.

The voice input device 410 may include a microphone configured toreceive a voice input, convert the voice input into an electric signal,and output the electric signal.

The non-voice input device 420 may include an input button, a touchscreen, or a camera provided in the mobile device 400.

The output device 430 may include a display, a speaker, or a vibratorprovided in the mobile device 400.

The voice input device 410, the non-voice input device 420, and theoutput device 430 provided as an input and output interface for a userin the mobile device 400 may be used, and the voice input device 210,the non-voice input device 220, and the dialogue partner output device230 provided in the vehicle 200 may also be used.

When the vehicle 200 transmits data detected by the vehicle monitor 260and a user input to the mobile device 400, the dialogue system client470 of the mobile device 400 delivers the data and the user input to theremote dialogue system server 1.

The dialogue system client 470 may deliver a response or a commandtransmitted from the remote dialogue system server 1 to the vehicle 200.When the dialogue partner output device 230 provided as an input andoutput interface for a user in the vehicle 200 is used, a response tospeech of the user may be output via the dialogue partner output device230. When the output device 430 provided in the mobile device 400 isused, a response to speech of the user may be output via the outputdevice 430 of the mobile device 400.

A command for vehicle control may be delivered to the vehicle 200 andthe controller 240 may execute control corresponding to the deliveredcommand to supply a service needed by the user.

On the other hand, the dialogue system client 470 may collect input dataand deliver the input data to the remote dialogue system server 1 andmay also execute some or all of the functions of the input processor 110and the result processor 130 of the dialogue system 100.

Referring to FIG. 16, the communication device 480 of the mobile device400 may include one or more communication modules configured forcommunicating with an external device and may include, for example, ashort-range communication module 481, a wired communication module 482,and a wireless communication module 483.

The short-range communication module 481 may include at least one ofvarious short-range communication modules configured to transmit andreceive signals at a short distance using a wireless communicationnetwork including a Bluetooth module, an infrared communication module,an RFID communication module, a WLAN communication module, an NFCcommunication module, and a Zigbee communication module.

The wired communication module 482 may include at least one of not onlyvarious wired communication modules including a LAN module, a WANmodule, and a VAN module but also various cable communication modulesincluding USB, HDMI, DVI, RS-232, power line communication, and POTS.

The wireless communication module 483 may include at least one of notonly a WiFi module and a WiBro module but also various wirelesscommunication modules connectable to an Internet network in a wirelesscommunication scheme including GSM, CDMA, WCDMA, UMTS, TDMA, and LTE,4G, and 5G.

For example, the mobile device 400 may be connected to the vehicle 200via the short-range communication module 481 or the wired communicationmodule 482 and may be connected to the remote dialogue system server 1or the external content server 300 via the wireless communication module483.

FIG. 17 is a control block diagram illustrating a mobile-independentscheme in which the dialogue system is mounted on a mobile device.

In the mobile-independent scheme, as illustrated in FIG. 17, thedialogue system 100 is provided in the mobile device 400.

Accordingly, it is not necessary to connect the mobile device 400 to theremote dialogue system server 1 to execute dialogue processing, and themobile device 400 may autonomously process a dialogue with the user andsupply a needed service to the user. Here, information necessary fordialogue processing and service supply may be taken from the externalcontent server 300.

Hereinafter, a detailed configuration of the dialogue system 100 and aspecific operation of each constituent element will be described morespecifically. In an exemplary embodiment to be described below, a casein which the dialogue system 100 is provided in the vehicle 200 will beassumed to facilitate the description.

FIG. 18, FIG. 19A, and FIG. 19B are control block diagrams illustratinga subdivided configuration of an input processor in the configuration ofthe dialogue system.

Referring to FIG. 18, the input processor 110 may include a voice inputprocessor 111 configured to process a voice input and a situationinformation processor 112 configured to process situation information.

A user's voice input via the voice input device 210 is transmitted tothe voice input processor 111, and a non-voice user input input via thenon-input device 220 is transmitted to the situation informationprocessor 112.

The controller 240 transmits the vehicle state information, the drivingenvironment information, and the user information to the situationinformation processor 112. The driving environment information and theuser information may be supplied from the external content server 300 orthe mobile device 400 connected to the vehicle 200.

The non-voice input may be included in the situation information. Thatis, the situation information may include vehicle state information,driving environment information, and user information.

The vehicle state information may include information indicating a stateof the vehicle as information acquired by detectors provided in thevehicle 200 and information stored as information regarding the vehicleincluding a type of fuel of the vehicle, in the vehicle.

The driving environment information may include video informationacquired by a front camera, a rear camera, or a stereo camera, obstacleinformation acquired by a detector including a radar, a lidar, or anultrasonic detector, and rainfall amount/rainfall speed informationacquired by a rainfall detector as information acquired by detectorsprovided in the vehicle 200.

The driving environment information may further include trafficsituation information, traffic light information, and informationregarding approach or collision likelihood of a nearby vehicle asinformation acquired through V2X.

The user information may include information regarding a state of theuser measured through a camera or a biometric signal measurement deviceprovided in the vehicle, information regarding the user directly inputusing an input device provided in the vehicle, information regarding theuser stored in the external content server 300, and information storedin the mobile device 400 connected to the vehicle.

The voice input processor 111 may include a voice recognizer 111 aconfigured to recognize an input user's voice and output the user'svoice as a spoken sentence in a text format, a natural languageunderstanding unit 111 b configured to ascertain an intention of theuser contained in the spoken sentence by applying a natural languageunderstanding technology to the spoken sentence, and a dialogue inputmanager 111 c configured to deliver a natural language understandingresult and situation information to the dialogue manager 120.

The voice recognizer 111 a may include a speech recognition engine. Thespeech recognition engine may recognize a voice spoken by the user byapplying a voice recognition algorithm to an input voice and generate arecognition result.

At the present time, since the input voice is converted into a formatmore useful for voice recognition, a starting point and an ending pointare detected from a voice signal to detect an actual voice sectionincluded in the input voice. This is referred to as end point detection(EPD).

As such, it is possible to extract a feature vector of the input voiceby applying a feature vector extraction technology including cepstrum,linear predictive coefficient (LPC), Mel frequency cepstral coefficient(MFCC), or filter bank energy, in the detected section.

As such, it is possible to obtain a recognition result by comparing theextracted feature vector to a reference pattern. To obtain therecognition result, an acoustic model configured to model signalcharacteristics of a voice for comparison and a language modelconfigured to model a language order relation between words, syllables,or the like corresponding to a recognized vocabulary may be used. To usethe acoustic model and the language model, an acoustic model/languagemodel DB may be stored in the storage 140.

The acoustic model may be classified into a direct comparison method ofsetting a recognition target with a feature vector model again andcomparing the recognition target to feature vectors of voice data and astatistical method of statistically processing feature vectors of arecognition target for use.

As the direct comparison method, there is a vector quantization methodas a representative method of setting a unit such as a word, a syllable,or the like which is a recognition target with a feature vector modeland comparing how an input voice is similar to the unit. The vectorquantization method is a method of mapping feature vectors of inputvoice data to a codebook which is a reference model and encoding thefeature vectors with representative values to compare the encodedvalues.

The statistical model method is a method of configuring a unit for arecognition target in state sequences and using a relation between thestate sequences. The state sequences may be configured as a plurality ofnodes. As a method of using the relation between the state sequences,there is a method in which dynamic time warping (DTW), a hidden Markovmodel (HMM), or a neuron network is used.

The DTW is a method of compensation for a difference on a time axis atthe time of a reference model in consideration of dynamiccharacteristics of a voice in which the length of a signal differs overtime even when the same person speaks with the same pronunciation. TheHMM is a recognition technology for assuming a voice by a Markov processhaving a state transition probability and an observation probability ofa node (output symbol) in each state and subsequently estimating thestate transition probability and the observation probability of the nodethrough learned data and determining a probability of a voice inputoccurring in an estimation model.

On the other hand, in a language model configured to model a languageorder relation between words, syllables, or the like, it is possible toreduce acoustic ambiguity and reduce a recognition error by applying anorder relation between units of language to units obtained in voicerecognition. As the language model, there are a statistical languagemodel and a model based on finite state automata (FSA). In thestatistical language model, a word series probability including unigram,bigram, trigram, or the like is used.

The voice recognizer 111 a may use any scheme among the above-describedschemes in recognition of a voice. For example, an acoustic model towhich the hidden Markov model is applied may be used or an N-bestsearching method in which an acoustic model and a voice model areintegrated may be used. In the N-best searching method, up to Nrecognition result candidates may be selected using the acoustic modeland the language model, and subsequently recognition performance may beimproved by reevaluating an order of the recognition result candidates.

The voice recognizer 111 a may determine a confidence value to guaranteeconfidence of the recognition result. The confidence value is an indexindicating how reliable a voice recognition result is. For example, theconfidence value can be defined as a relative value of a probability ofspeech of a syllable or a word which is a recognized result to aprobability of speech of another syllable or word. Accordingly, theconfidence value may be expressed as a value between 0 and 1 and mayalso be expressed as a value between 0 and 100.

When a confidence value exceeds a predetermined threshold, therecognition result may be output to execute an operation correspondingto the recognition result. When the confidence value is equal to or lessthan the threshold, the recognition result may be rejected.

A spoken sentence in a text format which is a recognition result of thevoice recognizer 111 a is input to the natural language understandingunit 111 b.

The natural language understanding unit 111 b may ascertain an intentionof the user contained in the spoken sentence by applying a naturallanguage understanding technology. Accordingly, the user may input acommand through a natural dialogue and the dialogue system 100 may helpthe user to input a command through a dialogue or supply a serviceneeded by the user.

First, the natural language understanding unit 111 b executes morphemeanalysis of the spoken sentence in the text format. Morphemes, thesmallest units of a meaning, are the smallest meaningful elements whichmay not be further subdivided. Accordingly, in the morpheme analysis, aninput sentence string is changed into a morpheme string as a first stageof the natural language understanding.

The natural language understanding unit 111 b extracts a domain from aspoken sentence based on a morpheme analysis result. By facilitating thedomain to identify subjects of a language spoken by the user, forexample, domains indicating various subjects including a route guide,weather searching, traffic searching, schedule management, a refuelingguide, and air-conditioning control are built as a database.

The natural language understanding unit 111 b may recognize an entityname from the spoken sentence. The entity name is a unique name of apersonal name, a place name, an organization name, a time, a date, acurrency, or the like. Recognition of the entity name is a job ofidentifying the entity name in a sentence and determining a kind ofrecognized entity name. An important keyword may be extracted from asentence through the recognition of the entity name and the meaning ofthe sentence may be ascertained.

The natural language understanding unit 111 b may analyze a speech actcorresponding to the spoken sentence. The analysis of the speech act isperformed with a job of analyzing an intention of a user's speech toascertain an intention of speech regarding whether a user inquires,requests, responds, or simply expresses a feeling.

The natural language understanding unit 111 b extracts an actioncorresponding to a speech intention of the user. Based on informationincluding a domain, an entity name, and a speech act corresponding tothe spoken sentence, the speech intention of the user is ascertained andan action corresponding to the speech intention is extracted. The actionmay be defined by an object or an operator.

The natural language understanding unit 111 b may also extract a factorrelated to action execution. The factor related to the action executionmay be an effective factor directly necessary to execute an action ormay be an ineffective factor used to extract the effective factor.

For example, when a spoken sentence of the user is “Let's go to SeoulStation,” the natural language understanding unit 111 b may extract“navigation” as a domain corresponding to the spoken sentence andextract “road guide” as an action. The speech act corresponds to“request.”

“Seoul Station” which is an entity name corresponds to [factor:destination] related to the action execution, but a specific exit numberor GPS information may be necessary for the road guide through actualnavigation. In the instant case, [factor: destination: Seoul Station]extracted by the natural language understanding unit 111 b may be acandidate factor for searching for Seoul Station actually desired by theuser in several Seoul Station POIs.

The natural language understanding unit 111 b may also extract means forexpressing a mathematical relation between words or sentences as in aparse-tree.

The morpheme analysis result, the domain information, the actioninformation, the speech act information, the extracted factorinformation, the entity name information, and the parse-tree which areprocessing results of the natural language understanding unit 111 b aredelivered to the dialogue input manager 111 c.

The situation information processor 112 may include a situationinformation collector 112 a configured to collect information from thenon-voice input device 220 and the controller 240, a situationinformation collection manager 112 b configured to manage collection ofsituation information, and a situation understanding unit 112 cconfigured to understand a situation based on the natural languageunderstanding result and the collected situation information.

The input processor 110 may include a memory configured to storeprograms used to execute the above-described operations or operations tobe described below and a processor configured to execute the storedprograms. A single memory and a single processor may be provided. Whenpluralities of memories and processors are provided, the memories andthe processors may be integrated on one chip or may be physicallydivided.

The voice input processor 111 and the situation information processor112 included in the input processor 110 may be realized by the sameprocessor or may be realized by separate processors.

Hereinafter, how the constituent elements of the input processor 110process input data using information stored in the storage 140 will bedescribed with reference to FIGS. 19A and 19B.

Referring to FIG. 19A, the natural language understanding unit 111 b mayuse a domain/action inference rule DB 141 to execute domain extraction,entity name recognition, speech act analysis, and action extraction.

A domain extraction rule, a speech act analysis rule, an entity nameconversion rule, an action extraction rule, and the like may be storedin the domain/action inference rule DB 141.

Other information such as the non-voice user input, the vehicle stateinformation, the driving environment information, and the userinformation may be input to the situation information collector 112 a tobe stored in the situation information DB 142, the long-term memory 143,or the short-term memory 144.

For example, raw data detected by the vehicle detector 260 may be storedseparately with a detector type and a detector value in the situationinformation DB 142.

Data meaningful to the user, such as a current state of a user andpreference/tendency of the user or data used to determine the currentstate or the preference/tendency of the user, may be stored in theshort-term memory 144 and the long-term memory 143.

As described above, information usable for a long time due to guaranteeof permanence, such as telephone numbers, schedules, preferences,educational background, personality, occupation, and family-relatedinformation related to the user, may be stored in the long-term memory143. Information used for a short time period due to non-guarantee ofpermanence and uncertainty, such as a current/previous location, today'sschedule, previous dialogue content, dialogue participants, asurrounding situation, a domain, and a driver state, may be stored inthe short-term memory 144. According to kinds of data, the data may bestored in duplicate in two or more storage regions of the situationinformation DB 142, the short term memory 144, and the long-term memory143.

Information related to which permanence is determined to be guaranteedamong information stored in the short-term memory 144 may be sent to thelong-term memory 143.

Information to be stored in the long-term memory 143 may also beacquired using information stored in the short-term memory 144 or thesituation information DB 142. For example, a preference of a user may beacquired by analyzing destination information or dialogue contentaccumulated for a provided time and the acquired preference of the usermay be stored in the long-term memory 143.

The information to be stored in the long-term memory 143 may be acquiredusing the information stored in the short-term memory 144 or thesituation information DB 142 inside the dialogue system 100 or in aseparate external system.

In the former case, the information may be acquired in the memorymanager 135 of the result processor 130 to be described below. In theinstant case, data used to acquire meaningful information or permanentinformation including a preference or tendency of the user, among thedata stored in the short-term memory 144 or the situation information DB142 may be stored in a log file format in the long-term memory 143. Thememory manager 135 analyzes data accumulated for a provided time ormore, acquires permanent data, and stores the data in the long-termmemory 143 again. A position at which the permanent data is stored inthe long-term memory 143 may be different from a position at which thedata stored in the log file format is stored.

The memory manager 135 may also determine permanent data among the datastored in the short-term memory 144 and may move and store thedetermined data in the long-term memory 143.

When the information may be acquired in the separate external system, asillustrated in FIG. 19B, a data management system 800 including acommunication device 810, a storage 820, and a controller 830 may beused.

The communication device 810 receives the data stored in the situationinformation DB 142 or the short-term memory 144. The stored data may allbe transmitted to the communication device, or only meaningfulinformation including the preference or the tendency of the user or onlydata used to acquire permanent information may be selected andtransmitted. The received data is stored in the storage 820.

The controller 830 analyzes the stored data, acquires the permanentdata, and transmits the acquired data to the dialogue system 100 via thecommunication device 810 again. The transmitted data is stored in thelong term memory 143 of the dialogue system 100.

The dialogue input manager 111 c may deliver an output result of thenatural language understanding unit 111 b to the situation understandingunit 112 c and obtain situation information regarding the actionexecution.

The situation understanding unit 112 c may determine what the situationinformation regarding the action execution corresponding to a speechintention of the user is, with reference to action-based situationinformation stored in the situation understanding table 145.

FIG. 20A and FIG. 20B are views illustrating exemplary informationstored in a situation understanding table.

Referring to an example of FIG. 20A, situation information and situationinformation type related to the action execution may be stored by anaction in the situation understanding table 145.

For example, when a road guide is an action, a current position may benecessary as the situation information and the situation informationtype may be GPS information. When vehicle state inspection is an action,a movement distance may be necessary as situation information and thesituation information type may be an integer. When gas stationrecommendation is an action, a remaining fuel amount and a drivabledistance (distance to empty (DTE)) may be necessary as the situationinformation and the situation information type may be an integer.

According to an included embodiment, the user may request the dialoguesystem 1 to adjust a service level. When service level adjustment is anaction, the service level may be necessary as the situation informationand the situation information type may be an integer.

When the situation information regarding the action executioncorresponding to a speech intention of the user has already been storedin the situation information DB 142, the long-term memory 143, or theshort-term memory 144, the situation understanding unit 112 c takescorresponding information from the situation information DB 142, thelong-term memory 143, or the short-term memory 144 and delivers thecorresponding information to the dialogue input manager 111 c.

When the situation information regarding the action executioncorresponding to a speech intention of the user is not stored in thesituation information DB 142, the long-term memory 143, or theshort-term memory 144, the situation understanding unit 112 c requeststhe situation information collection manager 112 b to transmit necessaryinformation. The situation information collection manager 112 b causesthe situation information collector 112 a to collect the necessaryinformation.

The situation information collector 112 a may periodically collect data,may collect data when a specific event occurs, or may periodicallycollect data and additionally collect data when a specific event occurs.The situation information collector 112 a may also collect data when adata collection request is input from situation information collectionmanager 112 b.

The situation information collector 112 a collects necessaryinformation, stores the necessary information in the situationinformation DB 142 or the short-term memory 144, and transmits aconfirmation signal to the situation information collection manager 112b.

The situation information collection manager 112 b also transmits aconfirmation signal to the situation understanding unit 112 c. Thesituation understanding unit 112 c takes the necessary information fromthe situation information DB 142, the long-term memory 143, or theshort-term memory 144 and delivers the necessary information to thedialogue input manager 111 c.

As a specific example, when an action corresponding to a speechintention of the user is a road guide, the situation understanding unit112 c may retrieve the situation understanding table 145 and know thatthe situation information regarding the road guide is the currentposition.

When the current position has already been stored in the short-termmemory 144, the situation understanding unit 112 c takes the currentposition from the short-term memory 144 and delivers the currentposition to the dialogue input manager 111 c.

When the current position is not stored in the short-term memory 144,the situation information collection manager 112 b is requested totransmit the current position. As such, the situation informationcollection manager 112 b causes the situation information collector 112a to acquire the current position from the controller 240.

The situation information collector 112 a acquires the current position,stores the current position in the short-term memory 144, and transmitsa confirmation signal to the situation information collection manager112 b. The situation information collection manager 112 b also transmitsthe confirmation signal to the situation understanding unit 112 c. Assuch, the situation understanding unit 112 c takes current positioninformation from the short-term memory 144 and delivers the currentposition information to the dialogue input manager 111 c.

The dialogue input manager 111 c may deliver an output of the naturallanguage understanding unit 111 b and an output of the situationunderstanding unit 112 c to the dialogue manager 120 and manageduplicate inputs so that the inputs are not input to the dialoguemanager 120. At the present time, the output of the natural languageunderstanding unit 111 b and the output of the situation understandingunit 112 c may be combined into one output and the combined output maybe delivered to the dialogue manager 120 or may be independentlydelivered.

On the other hand, when data collected by the situation informationcollector 112 a satisfies a predetermined condition and a specific eventis determined to occur, the situation information collection manager 112b may transmit an action trigger signal to the situation understandingunit 112 c.

The situation understanding unit 112 c retrieves the situationunderstanding table 145 to search for the situation informationregarding the relevant event. When the searched situation information isnot stored, a request signal of the situation information is transmittedagain to the situation information collection manager 112 b.

As exemplified in FIG. 20B, the situation information regarding eventsand the situation information types may be stored by event in thesituation understanding table 145.

An event occurring according to an exemplary embodiment of the presentinvention may be service level adjustment. The service level adjustmentmay be determined in accordance with the situation information collectedin the situation understanding table 145.

As in the example described in FIG. 3, when the user inputs speechrelated to a road guide, the dialogue system 100 may determine the roadguide by an action. Therefore, the dialogue system 100 may determine theservice level adjustment by an association event as the event related tothe road guide.

When the service level adjustment is determined by the event, thesituation understanding table 145 may collect and store the situationinformation regarding the service level adjustment.

As the situation information regarding the service level adjustmentstored in the situation understanding table 145 according to an includedexample, there are multiple kinds of information including informationregarding whether the user is visiting the destination for the firsttime, information regarding whether the vehicle enters a safe drivingcaution section, accident information, and weather information.

The situation information may be stored with the situation informationtype in the situation understanding table 145. When the situationinformation is the information regarding whether the user is travelingon a road for the first time, the situation information type is storedas an integer. When the situation information is the informationregarding whether the vehicle enters the safe driving caution section,the situation information type may be stored as a GPS value. When thesituation information is the accident information or the weatherinformation, the situation information type may be stored as text.

On the other hand, the situation information which is not stored in thesituation understanding table 145 is managed by the situationinformation collection manager 112 b. When the situation informationregarding the service level adjustment is not stored in the situationunderstanding table 145, the situation information collection manager112 b collects necessary situation information via the situationinformation collector 112 a and transmits a confirmation signal to thesituation understanding unit 112 c. The situation understanding unit 112c takes the necessary situation information from the situationinformation DB 142, the long-term memory 143, or the short-term memory144 and delivers the necessary situation information to the dialogueinput manager 111 c along with the action information.

The dialogue input manager 111 c inputs the output of the situationunderstanding unit 112 c to the dialogue manager 120.

FIG. 21 is a control block diagram illustrating a subdividedconfiguration of a dialogue manager, FIG. 22 is a view illustratingexemplary information stored in an associated action DB, FIG. 23 is aview illustrating exemplary information stored in an action executioncondition DB, and FIG. 24 is a view illustrating exemplary informationstored in an action factor DB.

Referring to FIG. 21, the dialogue manager 120 may include a dialogueflow manager 121 configured to form a request to generate/delete/updatea dialogue or an action, a dialogue action manager 122 configured togenerate/delete/update the dialogue or the action in a response to therequest from the dialogue flow manager 121, an ambiguity resolver 123configured to resolve ambiguity of a situation and ambiguity of adialogue and ultimately clarify an intention of the user, a factormanager 124 configured to manage a factor necessary for actionexecution, an action priority determiner 125 configured to determinewhether the plurality of candidate actions are executed and determinepriority of the candidate actions, and an external information manager126 configured to manage an external content list and relevantinformation and manage factor information necessary for an externalcontent query.

The dialogue manager 120 may include a memory configured to storeprograms used to execute the above-described operations or operations tobe described below and a processor configured to execute the storedprograms. A single memory and a single processor may be provided. Whenpluralities of memories and processors are provided, the memories andthe processors may be integrated on one chip or may be physicallydivided.

The constituent elements included in the dialogue manager 120 may berealized by a single processor or may be realized by separateprocessors.

The dialogue manager 120 and the input processor 110 may also berealized by a single processor or may be realized by separateprocessors.

A natural language understanding result (an output of a natural languageunderstanding unit) and the situation information (an output of thesituation understanding unit) which are outputs of the dialogue inputmanager 111 a are input to the dialogue flow manager 121. The output ofthe natural language understanding unit 111 b also includes not onlyinformation including a domain and an action but also informationregarding speech content of the user including a morpheme analysisresult. The output of the situation understanding unit 112 c may includenot only the situation information but also an event determined by thesituation information collection manager 112 b.

The dialogue flow manager 121 retrieves whether there is a dialogue taskor an action task corresponding to an input from the dialogue inputmanager 111 a in a dialogue/action DB 147.

The dialogue/action DB 147 is a storage space for managing states ofdialogues and states of actions and may store dialogue states and actionstates of a currently progressing dialogue and action and reserveactions which progress in the future. For example, states of an endeddialogue/action, a stopped dialogue/action, a progressingdialogue/action, and a dialogue/action which will progress may bestored.

A final output state of switching/nesting of an action, a switchedaction index, an action change time, a screen/voice/command, or the likemay be stored.

For example, when a domain and an action corresponding to a user'sspeech is extracted and there is a dialogue or an action correspondingto the domain and the action among the recently stored dialogues, thedialogue or the action may be determined to be a dialogue task or anaction task corresponding to an input from the dialogue input manager111 a.

When the domain and the action corresponding to the user's speech arenot extracted, the dialogue action generator 122 may be requested togenerate any task or refer to the recently stored task.

When there is no dialogue task or action task corresponding to theoutput of the input processor 110 in the dialogue/action DB 147, thedialogue flow manager 121 requests the dialogue action manager 122 togenerate a new dialogue task and a new action task.

The dialogue flow manager 121 may refer to a dialogue policy DB 148 tomanage the flow of a dialogue. The dialogue policy DB 148 stores apolicy for developing a dialogue and stores a policy forselecting/starting/proposing/stopping/ending a dialogue specifically.

The dialogue policy DB 148 may also store a policy for a viewpoint or amethod of outputting a response by the system and may store a policy formaking a response in association with several services and a policy fordeleting an existing action and substituting the existing action withanother action.

For example, when the number of candidate actions is plural or thenumber of actions corresponding to an intention or situation of the useris plural (actions A and B), both a policy of generating a response totwo actions including “Execute action A and then execute action B?” atone time and a policy for generating a response to one action andsubsequently generating a separate response to action B including“Action A is executed” and “Execute action B?” are possible.

According to an included embodiment, the dialogue system 100 may executea road guide action including “Road guide is started” in a response tospeech of a user making a request for guidance to Incheon InternationalAirport. Furthermore, the dialogue system 100 may output a result toinquire about whether an action of service level adjustment is executedin accordance with a dialogue policy.

The dialogue policy DB 147 may also store a policy for determiningpriority between candidate actions. The priority determination policywill be described later.

The dialogue action manager 122 allocates a storage space to thedialogue/action DB 147 and generates a dialogue task and an action taskcorresponding to the output of the input processor 110.

Conversely, when the domain and the action may not be extracted from thespeech of the user, the dialogue action manager 122 may generate anydialogue state. In the instant case, as will be described below, theambiguity resolver 123 may ascertain an intention of the user anddetermine an appropriate action corresponding to the intention of theuser based on speech content of the user, a surrounding situation, avehicle state, user information, and the like.

When there are the dialogue task and the action task corresponding tothe output of the input processor 110 in the dialogue/action DB 147, thedialogue flow manager 121 requests the dialogue action manager 122 torefer to the corresponding dialogue task and action task.

The action priority determiner 125 searches for an action listassociated with the action or the event included in the output of theinput processor 110 in the associated action DB 146 b to extract thecandidate actions.

As exemplified in FIG. 22, the associated action DB 146 b may indicatemutually associated actions, a relation between the associated actions,actions associated with events, and a relation between the events andthe associated actions. For example, at least one of the road guide, anaccident information guide, and a weather information guide may beclassified into an action associated with an action including servicelevel adjustment and a relation between the actions may correspond tomutual association.

For example, the accident information is accident information attractingattention of a driver while driving of the vehicle 200. When the driverexecutes a service supplied by the dialogue system 100 or participatesin a dialogue while driving the vehicle on a road on which there isaccident information, the driver may be distracted. Accordingly, thedialogue system 100 may classify actions of recommending the servicelevel adjustment while executing an action of providing the accidentinformation, and a relation between the actions may correspond to mutualassociation.

Specifically, in execution of an action of providing the accidentinformation, a service level adjustment action may be executed together.Here, executing the actions together may include both executing theservice level adjustment action before or after the action of providingthe accident information and executing the service level adjustmentaction together during the execution of the action of providing theaccident information.

When the event determined in the situation information collectionmanager 112 b is transmitted together from the input processor 110 inaddition to the action corresponding to the user's speech, both anaction associated with the action corresponding to the user's speech andaction associated with the event can become candidate actions.

The extracted candidate action list is delivered to the dialogue actionmanager 122. As such, the dialogue action manager 122 adds the candidateaction list to update an action state of the dialogue/action DB 147.

The action priority determiner 125 searches for conditions for executingeach candidate action in an action execution condition DB 146 c.

As exemplified in FIG. 23, the action execution condition DB 146 c maystore conditions necessary to execute actions and factors used todetermine whether the conditions are satisfied action by action.

For example, the service level adjustment action which is an associatedaction of the road guide action may include whether the user istraveling on a road for the first time, whether the vehicle on a currentroute enters a safe driving caution section, and whether collectedsituation information is accident information or weather information, asaction execution conditions.

The action execution condition DB 146 c may store a road guide history,GPS coordinates, a distance to a safe driving caution section, and areference level as condition determination factors along with theabove-described conditions.

The action priority determiner 125 delivers the execution conditions ofthe candidate actions to the dialogue action manager 122. As such, thedialogue action manager 122 adds the action execution conditioncandidate action by candidate action to update an action state of thedialogue/action DB 147.

The action priority determiner 125 may search for a factor necessary todetermine the action execution condition (hereinafter referred to as acondition determination factor) in the situation information DB 142, thelong-term memory 143, the short-term memory 144, or the dialogue/actionDB 147 and may determine whether each candidate action is executed usingthe searched factor.

When the factor used to determine the action execution condition is notstored in the situation information DB 142, the long-term memory 143,the short-term memory 144, or the dialogue/action DB 147, a necessaryfactor may be taken from the external content server 300 via theexternal information manager 126.

The action priority determiner 125 may determine whether each candidateaction is executed using the factor necessary to determine the actionexecution condition. The priority of each candidate action may bedetermined based on whether each candidate action is executed and apriority determination rule stored in the dialogue policy DB 148.

Referring back to FIG. 23, the action priority determiner 125 collectsthe road guide history as the condition determination factor related toa first trip. When a current road guide history is not found in the roadguide history, the action priority determiner 125 may determine that acondition for executing the service level adjustment action issatisfied.

The condition determination factor related to the entrance of the safedriving caution section may include current GPS coordinates and adistance to the safe driving caution section. When the distance from thecurrent position to the safe driving caution section is equal to orgreater than a predetermined distance, the action priority determiner125 may set the priority of the service level adjustment action to below since the vehicle has not yet entered the safe driving cautionsection.

According to the condition determination factor regarding whether theaccident information or the weather information is collected, a levelincluded in each piece of information may be determined. For example,each piece of accident information or weather information collected fromthe outside of the vehicle may include a level at which the dialoguesystem 100 can determine whether speech starts. When the accidentinformation collected as the situation information is a large-scaleaccident, the accident information may include a level valuepredetermined in various pieces of data. When the level value includedin the accident information is used as a condition determination factorand the level value exceeds a provided reference, the action prioritydeterminer 125 may determine that a condition for executing the servicelevel adjustment is satisfied.

A score for each candidate action may be determined in accordance with acurrent situation. Higher priority is allocated as the determined scoreis higher. For example, as expressed in [Math 1] below, an actioncorresponding a user's speech, a safety score, a convenience score, aprocessing time, a processing time point (whether a process isimmediately executed), user preference (the degree of accommodation ofthe user at the time of suggestion of a service or preference defined inadvance by the user), a manager score, a vehicle state associationscore, and an action success ratio (dialogue success ratio) may be usedas factors for score determination.Priority score=w1*user's speech action+w2*safety score+w3*conveniencescore+w4*processing time+w5*processing time point+w6*userpreference+w7*manager score+w8*vehicle state association score+w9*actionsuccess ratio*whether action is executable (1: executable, undecided, 0:unexecutable)*whether action is completed (completed: 1, uncompleted: 0)  [Math 1]

As described above, the action priority determiner 125 may supply aservice needed by the user by searching for not only actions directlyrelated to speech of the user or situation information but also anaction list associated with the actions and determining the priority ofthe actions.

The action priority determiner 125 delivers information regardingwhether each candidate action is executed and the priority to thedialogue action manager 122 and the dialogue action manager 122 adds thedelivered information to update an action state of the dialogue/actionDB 147.

The factor manager 124 may search for a factor used to execute eachcandidate action (hereinafter referred to as an action factor) in anaction factor DB 146 a.

As exemplified in FIG. 24, the action factor DB 146 a may store anessential factor, a selection factor, an initial value of a factor, anda reference position for taking a factor by action. When the initialvalue of the factor is stored, a factor value corresponding to arelevant factor is not included in speech of the user or situationinformation output from the input processor 110, and a relevant factorvalue is not stored in the situation information DB 142 either, anaction may be executed in accordance with the stored initial value orthe user may be allowed to confirm whether the action is executed inaccordance with the stored initial value.

For example, the essential factor used to execute a service leveladjustment action may include a level integer value and the selectionfactor may include situation information input through the inputprocessor 110. The initial value may include an integer value of acurrent service level.

The situation information may be acquired by sequentially retrieving thedialogue/action DB 147, the situation information DB 142, the short-termmemory 144, and the long-term memory 143. For example, the situationinformation regarding the reference position is used to take informationcollected by searching for the road guide history, the accidentinformation, and the weather information and determine whether theaction is executed.

As another example, the essential factor used to execute the road guideaction may include a current position and a destination and theselection factor may include a route type. A fast route may be storedwith the initial value of the selection factor. The current position andthe destination may be acquired by sequentially retrieving thedialogue/action DB 147, the situation information DB 142, the short-termmemory 144, and the long-term memory 143.

As described above, the factor manager 124 takes a factor value of afactor searched in the action factor DB 146 a from a relevant position.The reference position from which the factor value may be taken may beat least one of the situation information DB 142, the long-term memory143, the short-term memory 144, the dialogue/action DB 147, and theexternal content server 300.

When the factor manager 124 takes the factor value from the externalcontent server 300, the factor value may be taken via the externalinformation manager 126. The external information manager 126 maydetermine where information is taken with reference to an externalservice set DB 146 d.

The external service set DB 146 d stores information regarding anexternal content server associated with the dialogue system 100. Forexample, the external service set DB 146 d may store informationregarding an external service name, description of an external service,a type of information supplied by the external service, an externalservice using method, and an entity supplying the external service.

The factor value acquired by the factor manager 124 is delivered to thedialogue action manager 122. As such, the dialogue action manager 122adds a candidate-action-based factor value to the action state to updatethe dialogue/action DB 147.

The factor manager 124 may acquire the factor values of all thecandidate actions and may also acquire the factor value of the candidateaction determined to be executed by the action priority determiner 125.

The factor manager 124 may selectively use various factor valuesindicating the same information. For example, Seoul Station in a textformat indicating a destination may be converted into Seoul Station in aPOI format using a destination searching service of navigation.

When there is no ambiguity in a dialogue or a situation, necessaryinformation may be obtained through operations of the action prioritydeterminer 125, the factor manager 124, and the external informationmanager 126 described above, and the dialogue and the action may bemanaged. However, when there is ambiguity in the dialogue or thesituation, it is difficult to supply an appropriate service needed bythe user in only an operation of the action priority determiner 125, thefactor manager 124, and the external information manager 126.

In the instant case, the ambiguity resolver 123 may resolve theambiguity of the dialogue or the ambiguity of the situation. Forexample, when anaphora including “that person,” “that place fromyesterday,” “father,” “mother,” “grandmother,” and “daughter-in-law” isdisposed in a dialogue and there is ambiguity about what is indicated,the ambiguity resolver 123 may resolve such ambiguity with reference tothe situation information DB 142, the long-term memory 143, or theshort-term memory 144 and may suggest a guide for resolving suchambiguity.

For example, ambiguous expressions including “that place fromyesterday,” “Mart A near home,” and “Seoul Station where I wentyesterday” included in a dialogue may be factor values of action factorsor may correspond to factor values of condition determination factors.However, in the instant case, since the relevant words are ambiguous, anaction may not actually be executed or an action execution condition maynot be determined.

The ambiguity resolver 123 may resolve ambiguity of the factor valuewith reference to information stored in the situation information DB142, the long-term memory 143, or the short-term memory 144.Alternatively, necessary information may also be taken from the externalcontent server 300 using the external information manager 126 asnecessary.

For example, by searching for a place that the user visited yesterdaywith reference to the short-term memory 144, “that place from yesterday”may be converted into information which can be used as a destination ofthe road guide action. By searching for a home address of a user withreference to the long-term memory 143 and taking positional informationregarding Mart A near the home address from the external content server300, “Mart A near home” may be converted into information which can beused as the destination of the road guide action.

When an action (an object or an operator) is not clearly extracted inthe input processor 110 or an intention of the user is ambiguous, theambiguity resolver 123 may ascertain an intention of the user anddetermine a corresponding action with reference to an ambiguityresolution information DB 146 e.

FIG. 25 is a view illustrating exemplary information stored in anambiguity resolution information DB.

The ambiguity resolution information DB 146 e may store a spokensentence in association with an action corresponding to the spokensentence based on the vehicle state information and the surroundingsituation information. The spoken sentence stored in the ambiguityresolution information DB 146 e may be a spoken sentence from which anaction may not be extracted through natural language understanding. Inthe example of FIG. 25, a case in which speech content of a morphemeanalysis result is content the user's hand is cold or frozen will bedescribed.

The surrounding situation information may include vehicle-outsidetemperature and information regarding rain. The vehicle stateinformation may include information regarding whether an airconditioner/heater is turned on/off, an air volume, a wind direction,and information regarding whether a steering heat line is turned on/off.

As a specific example, when the outside temperature exceeds 20 degrees,it is raining, and an air conditioner is turned on, the temperature ofthe air conditioner may be set to be low, and thus it may be ascertainedthat the user's hands are cold. Thus, raising the temperature of the airconditioner by 3 degrees may be stored as a corresponding vehiclecontrol action.

When the outside temperature exceeds 20 degrees, it is raining, and anair conditioner is turned off, it may be ascertained that a user feelscold due to the rain. Thus, turning a heater on may be stored as acorresponding vehicle control action.

When the outside temperature exceeds 20 degrees, it is not raining, anair conditioner is turned on, and a wind direction of the airconditioner is set to be oriented upward, wind of the air conditioner isdirected toward the user's hands and it may be ascertained that theuser's hands are cold. Thus, changing a wind direction of the airconditioner to be oriented downward may be stored as a correspondingvehicle control action.

When the outside temperature exceeds 20 degrees, it is not raining, anair conditioner is turned on, a wind direction of the air conditioner isset to be oriented downward, and a wind amount is set to be medium orhigher, the wind direction of the air conditioner is strong and it maybe ascertained that the user is cold. Thus, lowering the wind amount ofthe air conditioner may be stored as a corresponding vehicle controlaction.

When the outside temperature exceeds 20 degrees, it is not raining, anair conditioner is turned on, a wind direction of the air conditioner isset to be oriented downward, and a wind amount is set to be weak (low),raising the temperature of the air conditioner by 3 degrees may bestored as a corresponding vehicle control action.

When the outside temperature is lower than 20 degrees and a heater isturned off, it may be ascertained that the user's hands are cold due tothe cold weather. Thus, turning the heater on may be stored as acorresponding vehicle control action.

When the outside temperature is lower than 20 degrees, a heater isturned on, and a steering heat line is turned off, it may be ascertainedthat the user's hands are cold because heat is not being delivered tothe hands. Thus, turning the steering heat line on may be stored as acorresponding vehicle control action.

When the outside temperature is lower than 20 degrees, both a heater anda steering heat line are turned on, and a wind direction of the heateris set to be oriented downward, the wind of the heater is not deliveredto the user's hands and thus it may be ascertained they are cold. Thus,changing the wind direction of the heater in both directions may bestored as a corresponding vehicle control action.

When the outside temperature is lower than 20 degrees, both a heater anda steering heat line are turned on, a wind direction of the heater isset to be oriented upward, and the temperature of the heater is set tobe lower than the maximum, raising the temperature of the heater may bestored as a corresponding vehicle control action.

When the outside temperature is lower than 20 degrees, both a heater anda steering heat line are turned on, a wind direction of the heater isset to be oriented upward, the temperature of the heater is set to bethe maximum, and the wind amount of the heater is not set to be themaximum, increasing the wind amount of the heater may be stored as acorresponding vehicle control action.

When the outside temperature is lower than 20 degrees, both a heater anda steering heat line are turned on, a wind direction of the heater isset to be oriented upward, both the temperature of the heater and thewind direction of the heater are set to be the maximum, and a heat lineseat is turned off, turning the heat line seat on may be stored as acorresponding vehicle control action.

When the outside temperature is lower than 20 degrees, both a heater anda steering heat line are turned on, a wind direction of the heater isset to be oriented upward, both the temperature of the heater and thewind amount of the heater are set to be the maximum, and a heat lineseat is turned on, an announcement to wait for a while since the heateris currently in full operation may be output in the correspondingvehicle control action.

FIG. 26A and FIG. 26B are tables summarizing various examples in whichthe ambiguity resolver resolves ambiguity, extracts an action withreference to the ambiguity resolution information DB, and control avehicle.

For example, as illustrated in FIG. 26A and FIG. 26B, when speechcontent of a morpheme analysis result is content in that the user'shands are cold or frozen, a surrounding situation is summer, a vehiclestate is a state in which a wind direction of an air conditioner isoriented toward the head of the user (upward), a predeterminedtemperature of the air conditioner is 19 degrees, and a wind amount isset to be the maximum, it may be ascertained that the user's hands arecold due to the wind of the air conditioner going to the hands.Accordingly, an air conditioner control action of changing the winddirection toward the user's feet (downward) and reducing the volume ofthe wind amount may be extracted as an action corresponding to thespeech and the vehicle may be controlled in accordance with theextracted action.

In the speech with the same content, when the surrounding situation iswinter, the vehicle state is a state in which the wind direction of theheater is oriented toward the feet of the passenger, a predeterminedtemperature of the air conditioner is 25 degrees, and a wind amount isset to be strong, it may be ascertained that the user's hands are coldbecause heat is not being delivered to the hands of the user.Accordingly, an action of turning a heat line of a steering wheel on maybe extracted as an action corresponding to the speech and the vehiclemay be controlled in accordance with the extracted action.

When the speech content of the morpheme analysis result is content thata user feels stuffy and the vehicle state is a state in which a vehiclespeed is 30 km/h or less and intervals before and after the vehicle areless than 30 cm, it may be ascertained that the user feels stuffy due totraffic congestion. Accordingly, an action of changing a route option(fast route guidance) in a road guide action, playing multimedia contentincluding music, or turning on a chatting function may be extracted asan action corresponding to the corresponding speech, and the vehicle maybe controlled in accordance with the extracted action.

When the speech content of the morpheme analysis result is content thata user feels drowsy and the vehicle state is a state of an internal-airmode, it may be ascertained that the drowsiness is caused due tonon-circulation of air. Accordingly, an action of changing theinternal-air mode to an outer-air mode may be extracted as an actioncorresponding to the speech and the vehicle may be controlled inaccordance with the extracted action.

In the speech with the same content, when the vehicle state is a statein which the outer-air mode is set and a heater is turned on, it may beascertained that the drowsiness is caused due to heat released from theheater. Accordingly, opening a window may be extracted as an actioncorresponding to the speech and the vehicle may be controlled inaccordance with the extracted action.

When the speech content of the morpheme analysis result is content thata user is sweating or feels hot, the surrounding situation is winter anda vehicle state is a state in which the heater is turned on, it may beascertained that the hotness is caused due to heat of the heater.Accordingly, lowering the temperature of the heater or reducing a windamount may be stored as an action corresponding to the speech.

In the speech with the same content, when the surrounding situation iswinter and the heater is turned off, it may be ascertained that thehotness is caused due to heat of the body of the user. Accordingly,opening a window or recommending opening a window may be extracted as anaction corresponding to the speech and the vehicle may be controlled inaccordance with the extracted action.

In the speech with the same content, when the surrounding situation issummer and the vehicle state is a state in which the air conditioner isturned off, it may be ascertained that the hotness is caused due to ahigh temperature inside the vehicle. Accordingly, turning the airconditioner on may be extracted as an action corresponding to the speechand the vehicle may be controlled in accordance with the extractedaction.

In the speech with the same content, when the surrounding situation issummer and the vehicle state is a state in which the air conditioner isturned on, it may be ascertained that the hotness is due to thetemperature of the air conditioner being set too high. Accordingly,lowering the temperature of the air conditioner or increasing a windamount may be extracted as an action corresponding to the speech and thevehicle may be controlled in accordance with the extracted action.

When the speech content of the morpheme analysis result is content thata user feels cold, the surrounding situation is summer and the vehiclestate is a state in which the air conditioner is turned on, it may beascertained that the coldness is due to the temperature of the airconditioner being to be low or the wind of the air conditioner beingstrong. Accordingly, raising the temperature of the air conditioner orreducing a wind amount may be extracted as an action corresponding tothe speech and the vehicle may be controlled in accordance with theextracted action.

In the speech with the same content, when the surrounding situation issummer and the vehicle state is a state in which the air conditioner isturned off, it may be ascertained that the coldness is caused due to aproblem of a body condition of the user. Accordingly, turning the heateron or checking a biorhythm of the user may be extracted as an actioncorresponding to the speech and the vehicle may be controlled inaccordance with the extracted action.

In the speech with the same content, when the surrounding situation iswinter and the vehicle state is a state in which the heater is turnedon, it may be ascertained that the coldness is due to the temperature ofthe heater being set too low and a wind amount being weak. Accordingly,changing the internal-air mode to the outer-air mode or opening a windowmay be extracted as an action corresponding to the speech and thevehicle may be controlled in accordance with the extracted action.

In the speech with the same content, when the surrounding situation iswinter and the vehicle state is a state in which the heater is turnedoff, it may be ascertained that the coldness is due to non-execution ofthe heater. Accordingly, executing the heater may be extracted as anaction corresponding to the speech and the vehicle may be controlled inaccordance with the extracted action.

When the speech content of the morpheme analysis result is content thata user has a headache, the surrounding situation is winter and thevehicle state is a state in which the heater is turned on, it may beascertained that the headache is caused due to non-circulation of air.Accordingly, raising the temperature of the air conditioner or reducinga wind amount may be extracted as an action corresponding to the speechand the vehicle may be controlled in accordance with the extractedaction.

In the speech with the same content, when the surrounding situation iswinter and the vehicle state is a state in which the heater is turnedoff, it may be ascertained that the headache is caused due to coldness.Accordingly, executing the heater may be extracted as an actioncorresponding to the speech and the vehicle may be controlled inaccordance with the extracted action.

In the speech with the same content, when the surrounding situation issummer and the vehicle state is a state in which the air conditioner isturned off, it may be ascertained that the headache is caused due tohotness. Accordingly, executing the air conditioner may be extracted asan action corresponding to the speech and the vehicle may be controlledin accordance with the extracted action.

In the speech with the same content, when the surrounding situation issummer and the vehicle state is a state in which the air conditioner isturned on, it may be ascertained that headache is caused due toair-conditioning. Accordingly, changing a wind direction or a windamount of the air conditioner may be extracted as an actioncorresponding to the speech and the vehicle may be controlled inaccordance with the extracted action.

When the speech content of the morpheme analysis result is content thata user feels uncomfortable, the surrounding situation is winter and itis raining, it may be ascertained that the discomfort is caused due tohigh humidity. Accordingly, executing a defogging function or executinga dehumidification function may be extracted as an action correspondingto the speech and the vehicle may be controlled in accordance with theextracted action.

In the speech with the same content, when the surrounding situation issummer and it is not raining, it may be ascertained that the discomfortis caused due to seasonal characteristics and heat. Accordingly,executing the air conditioner at the minimum temperature may beextracted as an action corresponding to the speech and the vehicle maybe controlled in accordance with the extracted action.

In the speech with the same content, when the surrounding situation issummer and it is raining, it may be ascertained that the discomfort iscaused due to heat and high humidity. Accordingly, executing the airconditioner in a dehumidification mode may be extracted as an actioncorresponding to the speech and the vehicle may be controlled inaccordance with the extracted action.

In the operation of the ambiguity resolver 123 described above, anaction actually desired by the user or an action actually necessary forthe user may be accurately ascertained and supplied by considering thesurrounding situation information and the vehicle state informationalong with speech of the user even when there is ambiguity in the speechor a situation of the user.

The information regarding the action determined by the ambiguityresolver 123 may be delivered to the dialogue action manager 122. Assuch, the dialogue action manager 122 may update the dialogue/action DB147 based on the delivered information.

Even for the action determined by the ambiguity resolver 123, the actionpriority determiner 125 and the factor manager 124 determine the actionexecution conditions, determine priority, and execute an operation oftaking a factor value, as described above.

When the values acquirable by the current situation and the dialogue areall acquired among the factor values used to execute each action, thedialogue action manager 122 transmits a signal to the dialogue flowmanager 121.

Conversely, when a factor value necessary for the action execution orthe condition determination is not in the dialogue/action DB 147, theexternal content server 300, the long-term memory 143, the short-termmemory 144, and the situation information DB 142 and is acquirable via auser, the result processor 130 may also generate a dialogue response toinquire of the user about the factor value.

The dialogue flow manager 121 delivers a dialogue state and informationregarding an action corresponding to the top priority to the resultprocessor 130. Information regarding a plurality of candidate actionsmay also be delivered in accordance with a dialogue policy.

FIG. 27 is a control block diagram illustrating a subdividedconfiguration of the result processor.

Referring to FIG. 27, the result processor 130 includes a responsegeneration manager 131 configured to manage generation of a responsenecessary to execute an action input from the dialogue manager 120, adialogue response generator 132 configured to generate a text response,an image response, or an audio response in a response to a request ofthe response generation manager 131, a command generator 136 configuredto generate a command for vehicle control or a command for supplying aservice using external content in a response to a request of theresponse generation manager 131, a service editor 134 configured tosequentially or sporadically execute several services to supply aservice desired by the user and gather result values, an output manager133 configured to output the generated text response, image response, oraudio response or output the command generated by the command generatorand decide an output order when the number of outputs is plural, and amemory manager 135 configured to manage the long-term memory 143 and theshort-term memory 144 based on the output of the response generationmanager 131 and the output manager 133.

The result processor 130 may include a memory configured to storeprograms used to execute the above-described operations or operations tobe described below and a processor configured to execute the storedprograms. A single memory and a single processor may be provided. Whenpluralities of memories and processors are provided, the memories andthe processors may be integrated on one chip or may be physicallydivided.

The constituent elements included in the result processor 130 may berealized by the same processor or may be realized by separateprocessors.

The result processor 130, the dialogue manager 120, and the inputprocessor 110 may be realized by the same processor or may be realizedby separate processors.

Responses output to correspond to speech or situations of the user mayinclude a dialogue response, vehicle control, and external contentsupply. The dialogue response may have a format of a response with aninitial dialogue, an inquiry, and information supply and may be storedas a database in a response template 149.

The response generation manager 131 requests the dialogue responsegenerator 132 and the command generator 136 to generate a responsenecessary to execute an action determined by the dialogue manager 120.To generate the response, information regarding an action to be executedmay be transmitted to the dialogue response generator 132 and thecommand generator 136. The information regarding the action to beexecuted may include an action name and a factor value. To generate theresponse, the dialogue response generator 132 and the command generator136 may refer to a current dialogue state and action state.

The dialogue response generator 132 may extract a dialogue response formby retrieving the response template 149 and generate a dialogue responseby filling a factor value necessary for the extracted dialogue responseform. The generated dialogue response is delivered to the responsegeneration manager 131. When the factor value necessary for thegeneration of the dialogue response is not delivered from the dialoguemanager 120 or an instruction to use external content, the externalcontent may be supplied from the external content server 300 or may bedelivered in the long-term memory 143, the short-term memory 144, or thesituation information DB 142.

For example, when the action determined by the dialogue manager 120corresponds to a road guide, the response template 149 may be retrievedto extract “Guide is started,” as the dialogue response form.

According to a disclosed embodiment, when the dialogue manager 120determines service level adjustment through an associated action,situation information regarding the service level adjustment iscollected. When a collected route history to a guided destinationindicates that it is the user's first visit, the dialogue manager 120may extract “This is your first time on the present route. You shouldlimit your use of the dialogue service,” and “Would you still like touse it?” as a dialogue response form from the response template 149.

When a response to speech or a situation of the user includes thevehicle control or the external content supply, the command generator136 generates a command to execute the vehicle control or the externalcontent supply. For example, when an action determined by the dialoguemanager 120 is control of an air conditioning device, a window, a seat,AVN, or the like, the command generator 136 generates a command toexecute the control and delivers the command to the response generationmanager 131.

Alternatively, when it is necessary to supply external content in theaction determined by the dialogue manager 120, the command generator 136generates a command to receive the content from the external contentserver 300 and delivers the command to the response generation manager131.

When the number of commands generated by the command generator 136 isplural, the service editor 134 determines an order and a method ofexecuting the plurality of commands and delivers the order and themethod to the response generation manager 131.

The response generation manager 131 delivers the response delivered fromthe dialogue response generator 132, the command generator 136, or theservice editor 134 to the output manager 133.

The output manager 133 determines an output timing, an output order, anoutput position, or the like of the dialogue response generated by thedialogue response generator 132 and the command generated by the commandgenerator 136.

The output manager 133 outputs the response by transmitting the dialogueresponse generated by the response generator 132 and the commandgenerated by the command generator 136 at an appropriate timing, in anappropriate order, and to an appropriate output position. A text tospeech (TTS) response may be output through the speaker 232 and the textresponse may be output through the display 231. When the dialogueresponse is output in a TTS format, a TTS module provided in the vehicle200 may be used or the output manager 133 may include the TTS module.

The command may be transmitted to the controller 240 depending on acontrol target or may be transmitted to the communication device 280 tocommunicate with the external content server 300.

The response generation manager 131 may also deliver a responsedelivered from the dialogue response generator 132, the commandgenerator 136, or the service editor 134 to the memory manager 135.

The output manager 133 may also deliver a response output by the outputmanager 133 to the memory manager 135.

The memory manager 135 manages the long-term memory 143 and theshort-term memory 144 based on the content delivered from the responsegeneration manager 131 and the output manager 133. For example, based onthe generated and output dialogue response, the memory manager 135 mayupdate the short-term memory 144 by storing dialogue content between theuser and the system or may update the long-term memory 143 by storinginformation regarding the user acquired through dialogue with the user.

Of the information stored in the short-term memory 144, meaningful andpermanent information such as a tendency or preference of the user orinformation used to acquire the meaningful and permanent information mayalso be stored in the long-term memory 143.

Based on the vehicle control corresponding to the generated and outputcommand or an external content request, the preference of the user, thevehicle control history, or the like stored in the long-term memory 143may also be updated.

In the dialogue system 100 according to the above-described embodiment,an optimum service needed by the user may be supplied in considerationof various situations occurring inside a vehicle. Even when speech of auser is not input, the dialogue system 100 can determine a serviceneeded by the user based on collected situation information or driverinformation and can supply the service preemptively.

The dialogue system 100 according to a disclosed example can limit theservice level supplied by the dialogue system 100 and selectively supplya service when a current situation input with the situation informationis a situation to which a driver has to pay attention.

For example, when the service level is limited and subsequently dialoguesteps with the user exceed the predetermined number of times, thedialogue system 100 can output a warning response. The dialogue system100 can limit the searching function so that the driver can drivecarefully. An exemplary embodiment will be described later in FIG. 30and the like.

Furthermore, the dialogue system 100 can vary an evaluation reference ofthe vehicle state in accordance with a situation at the time of startinga vehicle and can preemptively supply feedback. A driving start timepoint is defined as a vehicle starting time, an electronic parking brake(EPB) cancellation time, a navigation destination setting time.Weighting values are provided to a vehicle state evaluation systemdetermining a driving function score and individual units and thevariable weighting values to be applied to individual units are changeddepending on situation elements. When a vehicle state is determined tobe problematic, a problem solution for the individual unit including arepair shop guide may be supplied.

The dialogue system 100 may determine whether a fuel amount is not toreach a destination at the time of starting the vehicle, mayautomatically add a favorite gas station of the user as a stop along aroute to the destination as feedback when the fuel amount is not enough,and may guide the user to the gas station. A gas station automaticallyadded as a stop in a response to the user may be changed.

Even when a current vehicle state does not indicate that a fuel amountis not enough, a refueling place or time may be preemptively supplied byconsidering a future schedule of the user, a main movement record, aremaining fuel amount, and the like.

Information regarding a physical state and sleeping record of a drivermay be acquired and starting a vehicle may be permitted conditionallybased on the acquired information. For example, when the physical stateand the sleeping record are recognized outside of the vehicle and thus arisk of a drowsiness is recognized, a driver may be encouraged not todrive. Alternatively, information regarding recommended driving hoursmay be supplied in accordance with the physical state and the sleepingrecord.

When a trigger indicating a risk of drowsiness repeatedly occurs, a riskof drowsiness may be detected, and a warning may be output in accordancewith the degree of detected risk or feedback for automatically changinga route (a rest area) may be supplied. The trigger indicating a risk ofdrowsiness may be acquired by manually measuring a state of a driver ora state of a vehicle, for example, when a reduction in a hearing rate isindicated, a distance to a preceding vehicle is equal to or greater thana reference distance, or a speed is equal to or less than a referencespeed. Alternatively, the trigger may actively be acquired through adialogue, for example, by inquiring of a driver and measuring a responsespeed of the driver.

When a user inputs speech expressing his or her feeling, a specificdomain or action may not be extracted from the speech of the user.However, the dialogue system 100 can ascertain an intention of the userand develop a dialogue using surrounding situation information, vehiclestate information, user state information, or the like. The presentexample can be executed by causing the ambiguity resolver 230 to resolveambiguity of the speech of the user, as described above.

FIG. 28 is a flowchart illustrating adjustment of a service level by thedialogue system according to an exemplary embodiment.

According to an exemplary embodiment of the present invention, thedialogue system 100 collects the situation information while driving ofa vehicle (500).

The situation information may include driving environment informationand various other kinds of information. For example, the situationinformation includes at least one of information regarding whether it isa driver's first time on a driving route to a road guide destination,accident information on the driving route, a safe driving cautionsection in which a driver has to drive carefully, and weatherinformation. Even when the vehicle 200 is not driving, the situationinformation may include various information interfering withconcentration of a user.

On the other hand, the situation information is collected via thecontroller 240, the non-voice input device 220, and the communicationdevice 280 by the situation information collector 112 a included in thedialogue system 100.

The situation information collected by the situation informationcollector 112 a is delivered to the situation understanding unit 112 cvia the situation information collection manager 112 b. The situationunderstanding unit 112 c analyzes the situation information (510).

The situation understanding unit 112 c examines an association relationbetween the collected situation information and a currently executeddialogue service.

For example, the situation understanding unit 112 c may determine anassociation relation between a road guide requested by the user and theservice level adjustment. The situation understanding unit 112 c mayanalyze a history of the collected road guides and derive a result whichis a first trip. When it is the user's first time visiting a guidancedestination, the situation understanding unit 112 c may determine thatthe current situation information is associated with the service leveladjustment. The situation understanding unit 112 c delivers the analyzedinformation to the dialogue manager 120, the dialogue flow manager 121(520).

The dialogue flow manager 121 may determine whether an output regardingthe service level adjustment is executed through the currentdialogue/action state with the user. When the dialogue flow manager 121determines that a dialogue regarding the service level adjustment isnecessary in terms of a current dialogue policy, the dialogue flowmanager 121 controls the dialogue action manager 122.

The dialogue action manager 122 determines the service level to bechanged under the control of the dialogue flow manager 121 (530).

The dialogue action manager 122 may determine the service level inaccordance with a determination reference stored in the dialogue/actionDB 147. A specific determination reference will be described later inFIG. 29.

When the dialogue manager 120 determines the service level, the dialogueflow manager 121 determines speech for informing of a change in theservice level through the dialogue policy 148.

The determined dialogue policy and the service level may be delivered tothe result processor 130. As such, when the dialogue policy and theservice level are delivered, the result processor 130 may output a TTSresponse through the speaker 232 and output a text response through thedisplay 231 (540).

Thereafter, when agreement of the user is input, the dialogue system 100outputs a service executed by the dialogue system 100 and a controlsignal delivered to the controller 240 in accordance with the determinedservice level. A specific description of the service limitation will bemade later in FIG. 30.

FIG. 29 is a view illustrating a determination reference by which aservice level is determined according to an exemplary embodiment.

Referring to FIG. 29, the situation information related to the drivingenvironment information may include navigation information, accidentinformation, and weather information.

First, when the navigation information is input as situationinformation, the dialogue system 100 may use information regardingwhether it is the user's first time on the current route and whether theguidance driving route is the safe driving section, as a determinationreference. When the input navigation information corresponds to thedetermination reference, a service determination parameter increases by+1.

Here, the service determination parameter is a numerical value (integer)for determining the service level. That is, the dialogue system 100 mayapply the determination reference of the currently input drivingenvironment information and determine the service level in accordancewith a total sum of the service determination parameters for each pieceof information.

As another example, when the accident information is input as drivingenvironment information, the dialogue system 100 may use informationregarding whether there is accident information on the route as adetermination reference. When the accident information is disposed inthe current driving route, the dialogue system 100 increases the servicedetermination parameter by +1.

As yet another example, when the weather information is input as thedriving environment information, the dialogue system 100 may determinewhether the weather information corresponds to rain/fog/snow or the likeor may use information regarding whether attention on the currentdriving route is necessary for weather as a determination reference.

On the other hand, determination references of the service level in thedialogue system 100 are not limited thereto and various determinationreferences are used. The above-described determination references may becombined to serve as determination references of the service level. Forexample, when the situation information includes information indicatingthat it is the user's first time on the driving route and accidentinformation is received together during the first trip, a total sum ofthe service determination parameters may increase.

FIG. 30 is a view illustrating an example of limitation of a serviceadjusted in accordance with a service level.

According to a disclosed example, the service level is classified into abasic dialogue type, attention stage 1, attention stage 2, and attentionstage 3.

The basic dialogue type is a stage at which all the services to besupplied to the dialogue system 100 are supplied and may include all ofvarious operations of executing a dialogue with a user and controllingthe controller 240 in accordance with a dialogue result.

The attention stage 1 is a stage at which higher caution than the basicdialogue type is necessary. At the attention stage 1, the dialoguesystem 100 may limit a service to the user checking his or her ownspeech.

The service for the user checking his or her own speech may be a servicefor transmitting text or an e-mail through the mobile device 400. Whiledriving, the user may request the dialogue system 100 to transmit textor an e-mail to the outside via the mobile device 400 or thecommunication device 280 connected to the vehicle 200. After thedialogue system 100 completes a service, the user may request a servicefor confirming a transmitted factor. The service may endanger the userby reducing his or her concentration on driving since the user has toread text. Accordingly, at the attention stage 1, the dialogue system100 may limit a checking service for confirming text after transmittingtext or an e-mail.

The dialogue system 100 may utilize a response for requesting attentionas a dialogue policy when the number of dialogue steps increases alongwith the above-described service limitation.

The dialogue system 100 may request a driver to pay attention byoutputting a warning response to the driver when the number of dialoguesteps exceeds a predetermined number of times. The attention request maybe stored in the dialogue policy DB 148 and may be output under thecontrol of the dialogue manager 120.

The attention stage 2 is a stage at which higher attention than theattention stage 1 is necessary. At the attention stage 2, not only theservice limited at the attention stage 1 but also a searching functionis limited. Since there is a large probability of the user confirming asearching result in the searching function, there is a danger ofdecreasing concentration on driving.

The searching function may include a function of searching forinformation via an Internet communication network through the dialoguesystem 100 or searching for various information stored in the controller240.

The dialogue system 100 may utilize a response for requesting attentionas a dialogue policy when the number of dialogue steps increases alongwith the service limitation as in the attention stage 1.

The attention stage 3 is a stage at which higher attention than theattention stage 2 is necessary. The dialogue system 100 may execute onlyan emergency dialogue with a user.

The emergency dialogue includes a rescue request and urgent warningcontrol of a vehicle. At the attention stage 3, most of the servicessupplied by the dialogue system 100 may be limited.

On the other hand, the service level is merely an example of the presentinvention and can be modified variously without limitation.

FIG. 31 is a flowchart illustrating adjustment of a service levelaccording to another exemplary embodiment of the present invention.

Referring to FIG. 31, the dialogue system 100 may determine a servicelevel and output a response relevant to the service level, as describedin FIG. 28 and the like (600).

The dialogue system 100 may continuously receive situation informationand a voice of a user while executing an operation in accordance withthe determined service level (610).

After the service level is adjusted, the dialogue system 100continuously receives the situation information and the voice of theuser via the input processor 110. That is, although the service level ofthe dialogue system 100 is limited, the kind of collected information isnot limited.

The dialogue system 100 may determine whether the service level isreadjusted through the collected driving environment information andvoice of the user even after the service level is adjusted (620).

For example, when the situation information provided as a determinationreference of the service limitation is cancelled, the dialogue system100 delivers a situation of a change in the situation information to theuser. The adjustment of the service level may be inquired again.

As another example, while driving, the user may cancel the limitation inaccordance with the determined service level. The user may request thedialogue system 100 to change the service level. After the naturallanguage understanding of the user of the dialogue system 100, theservice level determined in accordance with a factor may be changed.

That is, the included dialogue system 100 may actively or passivelychange the service level and supply a service suitable for a situationeven when the service level is limited.

The dialogue system 100 may ask the user to agree when the service levelis determined to be readjusted (630).

Such stages are executed in accordance with a policy stored in thedialogue policy DB. When there is a response of the user, the dialoguesystem 100 executes the service in accordance with the changed servicelevel (640).

The dialogue system 100 included in the stages may supply a servicesuitable for the situation of the user beyond the limitation that anintention of the user is ascertained in accordance with an existingstandard dialogue scenario, minimize attention distraction of the user,and offer safe driving and general safety.

The foregoing descriptions of specific exemplary embodiments of thepresent invention have been presented for purposes of illustration anddescription. They are not intended to be exhaustive or to limit theinvention to the precise forms disclosed, and obviously manymodifications and variations are possible in light of the aboveteachings. The exemplary embodiments were chosen and described toexplain certain principles of the invention and their practicalapplication, to enable others skilled in the art to make and utilizevarious exemplary embodiments of the present invention, as well asvarious alternatives and modifications thereof. It is intended that thescope of the invention be defined by the Claims appended hereto andtheir equivalents.

What is claimed is:
 1. A dialogue system comprising: an input processorconfigured to extract situation information including at least one ofvehicle state information regarding a state of a vehicle and drivingenvironment information regarding a driving environment of the vehicleor an action corresponding to speech of a user; a storage configured tostore the extracted situation information; a dialogue manager configuredto determine a service level to be supplied based on the situationinformation and determine an action of adjusting the determined servicelevel; and a result processor configured to generate speech to executethe determined action and generate a control signal of servicelimitation based on the determined service level, wherein the storagestores a dialogue policy to be executed in accordance with thedetermined service level, and the result processor generates a responsefor a result of the service limitation based on the dialogue policy andoutputs the response for notifying the user that a service is adjustedin accordance with the determined service level, wherein the dialoguemanager applies a determination reference of currently input drivingenvironment information and determines the service level based on adetermination parameter, wherein the dialogue manager increases thedetermination parameter when the currently input driving environmentinformation corresponds to the determination reference, and wherein thedialogue manager acquires situation information necessary to determinewhether the action corresponding to the determined service level isexecuted from the storage and determines the action of adjusting theservice level based on the acquired situation information.
 2. Thedialogue system according to claim 1, wherein the storage storessituation information associated with the extracted action, and theinput processor extracts an action of service level adjustmentassociated with an action corresponding to the speech.
 3. The dialoguesystem according to claim 2, wherein the input processor requests thevehicle to transmit situation information necessary to determine theservice level when situation information regarding the action foradjusting the service level associated with the action corresponding tothe speech is not stored in the storage.
 4. The dialogue systemaccording to claim 1, wherein the storage stores the determinationparameter for determining the service level.
 5. The dialogue systemaccording to claim 4, wherein the dialogue manager determines theservice level based on the determination parameter and the situationinformation.
 6. The dialogue system according to claim 1, wherein thedialogue manager determines an action of readjusting the adjustedservice level based on new situation information received by the inputprocessor.
 7. The dialogue system according to claim 6, wherein theresult processor generates speech to execute an action regarding achanged service level.
 8. The dialogue system according to claim 6,wherein the result processor generates a control signal of the vehiclebased on the changed service level.
 9. The dialogue system according toclaim 1, further including: a communication device configured to receivethe situation information from the vehicle and transmit a response tothe vehicle.
 10. The dialogue system according to claim 1, furtherincluding: a communication device configured to communicate with anexternal server, wherein the dialogue manager requests the externalserver to transmit a factor value when the situation information is notable to be acquired from the storage.
 11. The dialogue system accordingto claim 1, further including: a communication device configured toreceive the situation information from a mobile device connected to thevehicle and transmit a response and the control signal of the servicelimitation to the mobile device.
 12. The dialogue system according toclaim 1, wherein the determination parameter is a numerical value fordetermining the service level, and wherein the dialogue managerdetermines the service level in accordance with a total sum ofdetermination parameters for each of the driving environmentinformation.
 13. A dialogue service processing method comprising:storing situation information including at least one of vehicle stateinformation regarding a state of a vehicle and driving environmentinformation regarding a driving environment of the vehicle in a storage;extracting the situation information and an action corresponding tospeech of a user; acquiring, from the storage, at least one of a factorvalue of a condition determination factor used to determine whether atleast one action candidate including the action corresponding to thespeech is executed and a factor value of an action factor used toexecute the at least one action candidate; determining a service levelto be supplied based on the situation information and an action to beexecuted from the at least one action candidate based on the acquiredfactor value of the condition determination factor; and generating acontrol signal of service limitation based on the determined servicelevel and a response to execute the determined action using the acquiredfactor value of the action factor, wherein the storage stores a dialoguepolicy to be executed in accordance with the service level, and thegenerating of the control signal includes generating a response for aresult of the service limitation based on the dialogue policy andoutputting the response for notifying the user that a service isadjusted in accordance with the determined service level, wherein thedetermining the service level to be supplied based on the situationinformation comprises: applying a determination reference of currentlyinput driving environment information; and determining the service levelbased on a determination parameter, and wherein the determinationparameter increases when the currently input driving environmentinformation corresponds to the determination reference, and wherein thedetermining of the service level includes determining the service levelusing the situation information as the factor value of the conditiondetermination factor or the factor value of the action factor.
 14. Thedialogue service processing method according to claim 13, furtherincluding: storing situation information regarding an action in thestorage by action; and acquiring situation information regarding theaction corresponding to the speech from the storage and transmitting theacquired situation information to a dialogue manager.
 15. The dialogueservice processing method according to claim 14, further including:requesting the vehicle to transmit situation information necessary todetermine the service level when the situation information regarding theaction corresponding to the speech is not stored in the storage.
 16. Thedialogue service processing method according to claim 13, wherein theservice level is changed based on the situation information delivered bythe vehicle.
 17. The dialogue service processing method according toclaim 13, wherein the situation information includes at least one ofnavigation information, accident information, and weather information.18. The dialogue service processing method according to claim 13,wherein the determination parameter is a numerical value for determiningthe service level, and wherein the determining the service level inaccordance with a determination parameter comprises determining theservice level in accordance with a total sum of determination parametersfor each of the driving environment information.
 19. A vehiclecomprising: a dialogue system including: an input processor configuredto extract situation information including at least one of vehicle stateinformation regarding a state of a vehicle and driving environmentinformation regarding a driving environment of the vehicle or an actioncorresponding to speech of a user, a storage configured to store theextracted situation information, a dialogue manager configured todetermine a service level to be supplied based on the situationinformation and determine an action of adjusting the determined servicelevel, and a result processor configured to generate speech to executethe determined action and generate a control signal of servicelimitation based on the determined service level, wherein the storagestores a dialogue policy to be executed in accordance with thedetermined service level, and the result processor generates a responsefor a result of the service limitation based on the dialogue policy andoutputs the response for notifying the user that a service is adjustedin accordance with the determined service level, wherein the dialoguemanager applies a determination reference of currently input drivingenvironment information and determines the service level based on adetermination parameter, wherein the dialogue manager increases thedetermination parameter when the currently input driving environmentinformation corresponds to the determination reference, and wherein thedialogue manager acquires situation information necessary to determinewhether the action corresponding to the determined service level isexecuted from the storage and determines the action of adjusting theservice level based on the acquired situation information.
 20. Thevehicle according to claim 19, further including: a controllerconfigured to execute an operation related to the determined servicebased on the control signal.
 21. The vehicle according to claim 19,wherein the determination parameter is a numerical value for determiningthe service level, and wherein the dialogue manager determines theservice level in accordance with a total sum of determination parametersfor each of the driving environment information.